A Persistent State for Auditable Mixture-of-Experts Routing

Abdurrahman Javat; Allan Kazakov

cs.AISep 28, 2026

A Persistent State for Auditable Mixture-of-Experts Routing

Authors: Abdurrahman Javat, Allan Kazakov

Organizations: Department of Artificial Intelligence Bahçeşehir University Istanbul, Türkiye

Abstract

Mixture-of-Experts (MoE) models repeatedly route tokens to sparse subsets of experts, but conventional routers expose no routing-specific record of how cross-layer influences accumulate. We introduce Scratchpad-Augmented Mixture-of-Experts (SA-MoE), which gives each router access to a low-dimensional persistent state that is not provided to the experts. Learned layerwise writes update this state, and their realized post-update changes exactly decompose the state-mediated contribution to any later routing margin, forming a routing ledger. Across sparsely upcycled SmolLM2- and Gemma-based models and three independent training seeds per architecture, this pathway adds less than 1% analytical forward compute and is strongly used by trained routers: local removal of its router contribution changes the selected Top-2 expert set in 87.6% and 69.9% of decisions, respectively. Relative to a matched latest-write-only control, persistent accumulation increases long-horizon future-routing accessibility by 19.4 and 12.2 percentage points, with positive effects in every seed. More than 90% of absolute ledger contribution comes from non-recent writes in both families, and full-forward suppression of ledger-selected writes changes later routing and output distributions. The ledger is an exact provenance object for the persistent-state pathway, not a complete causal explanation of routing. Sensitivity-aware scores better predict full-forward intervention effects, and post-hoc methods recover related cross-layer attribution without architectural modification. SA-MoE instead makes one routing-specific computational history explicit and directly inspectable within the model's natural forward computation.

Figures & tables

Family	Architecture	Params	Active/token	FLOPs/token	$\Delta$ PPL (%)	$\Delta$ Macro (pp)
SmolLM2	Ordinary MoE	1.409B	294.0M	729.6M	Ref.	Ref.
	Latest-write-only	1.411B	296.3M	734.1M	$+0.52$ [ $+0.18,+0.86$ ]	$-0.02$ [ $-0.48,+0.43$ ]
	Persistent	1.411B	296.3M	734.1M	$+0.19$ [ $+0.12,+0.25$ ]	$-0.05$ [ $-0.14,+0.04$ ]
Gemma	Ordinary MoE	1.401B	409.8M	876.2M	Ref.	Ref.
	Latest-write-only	1.402B	411.4M	879.2M	$+0.30$ [ $-0.18,+0.79$ ]	$-0.49$ [ $-1.31,+0.33$ ]
	Persistent	1.402B	411.4M	879.2M	$-0.06$ [ $-0.45,+0.32$ ]	$-0.16$ [ $-1.66,+1.35$ ]

Table 1: Model quality and architectural cost across three training seeds. Quality columns are paired mean changes relative to Ordinary MoE; brackets give 95% Student-

t

intervals over training seeds (

n=3

). Macro is the unweighted mean of HellaSwag, ARC-Easy, PIQA, and WinoGrande primary metrics.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

	SmolLM2	Gemma
Dense source	SmolLM2-135M	Gemma-3-270M
Transformer layers	30	18
Hidden size	576	640
Expert intermediate size	1536	2048
Routed experts / layer	16	16
Shared experts / layer	1	1

Appendix

Table 2: Primary model and MoE configuration. The routing-state entries apply to Persistent and latest-write-only; Ordinary MoE has no routing state.

	SmolLM2	Gemma
Precision	bf16	bf16
Optimizer steps	30,000	30,000
Sequence length	2048	2048
World size	16	16
Global sequences / step	512	512
Tokens / optimizer step	1,048,576	1,048,576

Appendix

Table 3: Continued-pretraining configuration. Microbatch and accumulation differ where needed to fit the model family while preserving the same global batch and token budget.

Family	Architecture	Params.	Active/token	Fwd. FLOPs/token
SmolLM2	Ordinary	1.4088B	294.044M	729.575M
	Latest-write-only	1.4111B	296.321M	734.122M
	Persistent	1.4111B	296.321M	734.122M
Gemma	Ordinary	1.4007B	409.840M	876.192M
	Latest-write-only	1.4023B	411.354M	879.215M
	Persistent	1.4023B	411.354M	879.215M

Appendix

Table 4: Model size and analytical active compute. Parameter counts include each unique parameter once. Values are shown at higher precision than in the main-text table.

Family	Architecture	Inf. B1	Inf. B16	Train B4
SmolLM2	Ordinary	117.3	192.2	446.8
	Latest-write-only	151.4	210.7	545.8
	Persistent	140.9	204.2	524.4
Gemma	Ordinary	94.3	191.8	318.4
	Latest-write-only	82.2	176.9	326.6
	Persistent	82.5	182.3	316.4

Appendix

Table 5: Common-hardware median runtime in milliseconds on one NVIDIA GH200 120GB. B1/B16 denote inference batch size; Train B4 is a forward–backward training step with batch size 4. These timings are descriptive and are not used as statistical architecture claims.

Task	Split	Examples	Primary metric
HellaSwag	validation	10,042	normalized accuracy
ARC-Easy	test	2,376	normalized accuracy
PIQA	validation	1,838	normalized accuracy
WinoGrande	winogrande_xl validation	1,267	raw accuracy

Appendix

Table 6: Downstream capability evaluation protocol.

Family	Architecture	HellaSwag	ARC-Easy	PIQA	WinoGrande	Macro
SmolLM2	Latest-write-only	$+0.14$	$-0.36$	$+0.29$	$-0.16$	$-0.02$
	Persistent	$+0.01$	$-0.51$	$+0.36$	$-0.08$	$-0.05$
Gemma	Latest-write-only	$-0.39$	$-1.09$	$-0.33$	$-0.16$	$-0.49$
	Persistent	$-0.44$	$-0.36$	$+0.07$	$+0.11$	$-0.16$

Appendix

Table 7: Three-seed mean task-level capability changes relative to matched Ordinary MoE, in percentage points. Macro is the unweighted four-task mean. These are paired architecture contrasts, not differences between model families.

Family / source	Horizon 1	Horizon 4	Horizon 8
SmolLM2, source 7	$7\!\rightarrow\!8$	$7\!\rightarrow\!11$	$7\!\rightarrow\!15$
SmolLM2, source 14	$14\!\rightarrow\!15$	$14\!\rightarrow\!18$	$14\!\rightarrow\!22$
Gemma, source 4	$4\!\rightarrow\!5$	$4\!\rightarrow\!8$	$4\!\rightarrow\!12$
Gemma, source 8	$8\!\rightarrow\!9$	$8\!\rightarrow\!12$	$8\!\rightarrow\!16$

Appendix

Table 8: Frozen source–target layer pairs for future-routing prediction. Horizons count MoE-layer distance. Expert indices are local to each target router; equal numeric indices at different layers do not denote the same physical expert.

Family	Representation	10k	100k	1M
SmolLM2	Persistent native state	79.28	81.06	81.25
	Latest-write-only native state	56.87	58.55	58.73
	P hidden-128	77.76	82.21	83.37
Gemma	Persistent native state	74.95	77.15	77.41
	Latest-write-only native state	64.76	66.77	66.99
	P hidden-128	78.19	82.70	84.10

Appendix

Table 9: Three-seed mean transition-only future-routing accuracy (%) for the principal probe representations. “P hidden-128” is the supervised 128-dimensional subspace learned from the Persistent model’s ordinary hidden stream. Values are equal-pair means over all six frozen source–target pairs.

Family	Top-1 margin	Top-2 boundary
SmolLM2	$3.815\times 10^{-6}$	$3.785\times 10^{-6}$
Gemma	$2.623\times 10^{-6}$	$2.861\times 10^{-6}$

Appendix

Table 10: Numerical reconstruction of the Persistent ledger on the frozen construction checkpoints. Errors are maximum absolute residuals between the summed write contributions and the corresponding direct target-state margin term. This is a numerical validation of the descriptive decomposition, not an intervention result.

Family	Method	Spearman	Top-source acc.	Normalized regret
SmolLM2	Random	—	$0.073\,[0.021,0.125]$	$0.706\,[0.646,0.758]$
	Recency	$0.150\,[0.092,0.205]$	$0.188\,[0.115,0.271]$	$0.505\,[0.430,0.580]$
	Largest write norm	$0.187\,[0.132,0.241]$	$0.135\,[0.062,0.219]$	$0.499\,[0.437,0.562]$
	Ledger	$0.369\,[0.319,0.421]$	$0.250\,[0.156,0.344]$	$0.362\,[0.307,0.417]$
	Gradient $\times$ write	$0.769\,[0.723,0.807]$	$0.771\,[0.688,0.844]$	$0.055\,[0.032,0.081]$
Gemma	Random	—	$0.031\,[0.000,0.062]$	$0.742\,[0.687,0.792]$

Appendix

Table 11: Exhaustive full-forward ranking quality on the fixed Persistent seed-42 checkpoints. Entries are means with 95% paired sequence-bootstrap intervals over 24 source sequences and 96 targets. Higher Spearman and top-source accuracy are better; lower normalized oracle regret is better. Random supplies a selected-source baseline but no complete deterministic score vector for the SA-MoE Spearman calculation.

Family	Method	Spearman	Top-source acc.	Regret
SmolLM2	Random	$-0.030\,[-0.046,-0.014]$	$0.007\,[-0.023,0.037]$	$0.916\,[0.905,0.928]$
	Recency	$0.125\,[0.089,0.161]$	$0.090\,[0.051,0.130]$	$0.683\,[0.616,0.750]$
	Largest component norm	$0.662\,[0.638,0.687]$	$0.319\,[0.240,0.398]$	$0.503\,[0.429,0.578]$
	Li variance	$0.653\,[0.634,0.672]$	$0.090\,[0.060,0.120]$	$0.698\,[0.592,0.803]$
	Li Top-2 boundary	$0.570\,[0.542,0.599]$	$0.212\,[0.172,0.251]$	$0.581\,[0.510,0.652]$
	Gradient $\times$ component	$0.721\,[0.670,0.771]$	$0.326\,[0.154,0.499]$	$0.404\,[0.278,0.530]$

Appendix

Table 12: Ordinary-MoE attribution ranking across training seeds 42/43/44. Entries are three-seed means with 95% Student-

t

intervals (

n=3

, df

=2

). Higher Spearman and top-source accuracy are better; lower regret is better.

Family	Audit object	Capture (s)	Capture peak (GiB)	Post-capture score (s)	Gradient extraction (s)
SmolLM2	Persistent ledger	0.249	3.96	included	3.205
	Ordinary Li components	0.752	2.85	0.152	1.146
Gemma	Persistent ledger	0.428	10.88	included	1.260
	Ordinary Li components	0.852	10.70	0.080	1.319

Appendix

Table 13: Phase-isolated attribution-cost pilot. “Capture” is one clean forward with the minimal representation required by the corresponding audit object. Peak is maximum allocated accelerator memory during that capture. For the native ledger, contribution materialization occurs during capture and requires zero incremental model passes. Li score extraction operates on captured components and likewise uses zero incremental forwards or backwards. Gradient extraction is shown separately because it is target dependent.

Family	Architecture	Target $\|\Delta m^{(2,3)}\|$	Downstream Top-2 divergence	Output KL
SmolLM2	Persistent	$0.276\,[0.156,0.396]$	$0.080\,[0.040,0.120]$	$0.00155\,[0.00115,0.00195]$
	Latest-write-only	$0.180\,[-0.057,0.417]$	$0.030\,[0.010,0.051]$	$0.00159\,[0.00107,0.00211]$
	P–LWO	$0.096\,[-0.179,0.371]$	$0.050\,[-0.009,0.108]$	$-0.00004\,[-0.00064,0.00056]$
Gemma	Persistent	$0.426\,[0.338,0.515]$	$0.195\,[0.126,0.264]$	$0.01615\,[0.00521,0.02709]$
	Latest-write-only	$0.279\,[0.151,0.407]$	$0.049\,[0.003,0.095]$	$0.00609\,[0.00063,0.01156]$
	P–LWO	$0.147\,[-0.034,0.328]$	$0.146\,[0.041,0.251]$	$0.01006\,[-0.00510,0.02521]$

Appendix

Table 14: Full-forward source-suppression outcomes across training seeds. Entries are three-seed means with 95% Student-

t

intervals (

n=3

, df

=2

). The P–LWO row is the paired Persistent-minus-latest-write-only contrast.

Family	Architecture	Seed	PPL ratio	$\Delta$ Hella	$\Delta$ ARC-E	$\Delta$ PIQA	$\Delta$ Wino	$\Delta$ Macro
SmolLM2	Latest-write-only	42	1.00361	$-0.159$	$-0.505$	$-0.054$	$+0.158$	$-0.140$
		43	1.00557	$+0.309$	$-0.715$	$+0.054$	$+1.105$	$+0.188$
		44	1.00625	$+0.259$	$+0.126$	$+0.871$	$-1.736$	$-0.120$
	Persistent	42	1.00213	$-0.020$	$-0.210$	$+0.326$	$-0.158$	$-0.015$
		43	1.00187	$+0.020$	$-1.389$	$+0.381$	$+0.631$	$-0.089$
		44	1.00161	$+0.030$	$+0.084$	$+0.381$	$-0.710$	$-0.054$

Appendix

Table 15: Seed-level quality changes relative to matched Ordinary MoE. HellaSwag, ARC-Easy, and PIQA use normalized accuracy; WinoGrande uses raw accuracy. Macro is their unweighted mean.

Family	$\lambda$	Seed 42	Seed 43	Seed 44
SmolLM2	0.75	11.89	12.34	11.37
	0.50	28.88	30.22	27.10
	0.25	54.25	56.48	51.61
	0	88.16	88.61	86.17
Gemma	0.75	17.88	17.03	15.94
	0.50	34.67	35.76	33.91

Appendix

Table 16: Persistent selected-Top-2 set-change rate under local state attenuation, in percent. The intervention holds the natural hidden representation and state fixed and changes only the current router’s direct state contribution.

Family	Endpoint	Seed 42	Seed 43	Seed 44	Mean [95% CI]
SmolLM2	Long-horizon accessibility (pp)	21.97	19.33	16.99	$19.43\,[13.24,25.62]$
	Ledger–control advantage	0.143	0.060	$-0.009$	$0.065\,[-0.124,0.253]$
	Ledger intervention effect	0.174	0.124	0.016	$0.105\,[-0.096,0.305]$
Gemma	Long-horizon accessibility (pp)	9.68	15.09	11.75	$12.17\,[5.40,18.94]$
	Ledger–control advantage	0.046	0.234	0.040	$0.106\,[-0.167,0.380]$
	Ledger intervention effect	0.059	0.257	0.086	$0.134\,[-0.134,0.402]$

Appendix

Table 17: Seed-level Persistent-minus-latest-write-only persistence endpoints. Long-horizon accessibility is in percentage points; the two intervention endpoints are in target-margin units. The final column gives the three-seed mean and Student-

t

95% interval.

Family	Budget	Representation	Seed 42	Seed 43	Seed 44
SmolLM2	10k	Persistent native	79.28	78.37	80.18
		Latest-write-only native	57.90	56.24	56.48
		P hidden-128	76.56	77.93	78.81
	100k	Persistent native	81.18	79.88	82.12
		Latest-write-only native	59.50	57.76	58.40
		P hidden-128	81.52	82.20	82.92

Appendix

Table 18: Seed-level transition-only future-routing accuracy (%) underlying Figure 3 . “P hidden-128” denotes the supervised 128-dimensional representation learned from the Persistent model’s hidden stream.

Family	Metric	Seed 42	Seed 43	Seed 44	Mean [95% CI]
SmolLM2	Non-recent absolute share (%)	91.24	92.20	91.76	$91.73\,[90.55,92.92]$
	Effective source support	9.444	9.544	9.482	$9.490\,[9.364,9.616]$
	Largest absolute source share (%)	20.57	20.10	20.47	$20.38\,[19.77,21.00]$
	Signed direct margin	0.478	0.343	0.403	$0.408\,[0.239,0.577]$
Gemma	Non-recent absolute share (%)	90.75	89.84	90.28	$90.29\,[89.15,91.43]$
	Effective source support	5.776	5.923	5.774	$5.824\,[5.613,6.036]$

Appendix

Table 19: Seed-level Persistent direct-provenance summaries. Contribution-share columns use absolute selected-set-margin contribution. Signed margin is the net direct state-mediated selected-set-margin contribution.

Family	Method	Metric	Seed 42	Seed 43	Seed 44
SmolLM2	Li Top-2 boundary	Spearman	0.577	0.557	0.576
		Top-source accuracy	0.208	0.198	0.229
		Regret	0.575	0.612	0.555
	Gradient $\times$ component	Spearman	0.731	0.697	0.734
		Top-source accuracy	0.281	0.292	0.406
		Regret	0.413	0.450	0.350

Appendix

Table 20: Seed-level Ordinary-MoE attribution-ranking results for the Li Top-2-boundary adaptation and gradient

\times

component. Higher Spearman and top-source accuracy are better; lower regret is better.

Family	Architecture / contrast	Metric	Seed 42	Seed 43	Seed 44
SmolLM2	Persistent	Target $\|\Delta m\|$	0.3289	0.2344	0.2641
		Downstream Top-2	0.0808	0.0958	0.0638
		Output KL	0.001386	0.001709	0.001556
	Latest-write-only	Target $\|\Delta m\|$	0.1365	0.1138	0.2891
		Downstream Top-2	0.0257	0.0257	0.0400
		Output KL	0.001442	0.001504	0.001832

Appendix

Table 21: Seed-level full-forward source-suppression outcomes. Target

|\Delta m|

is the absolute change in the fixed clean Top-2 boundary; downstream Top-2 is mean Jaccard distance; KL is

D_{\mathrm{KL}}(p_{\mathrm{clean}}\|p_{\mathrm{int}})

Analysis	Checkpoint scope	Uncertainty / replication unit	Interpretation
Model quality and capability	Seeds 42/43/44 for each family and architecture	Training seed; Student- $t$ ( $df=2$ ). PPL analyzed on log-ratio scale.	Replicated architecture-level result.
Local state-suppression dose response	Persistent and latest-write-only, seeds 42/43/44	Training seed; Student- $t$ ( $df=2$ )	Replicated local router-dependence result.
Long-horizon persistence endpoint	Persistent and latest-write-only, seeds 42/43/44	Paired training seed; Student- $t$ ( $df=2$ )	Primary replicated persistence result.
Main future-routing probe curves	Seeds 42/43/44	Training seed after within-checkpoint fit/pair reduction; Student- $t$ ( $df=2$ )	Replicated representation-accessibility result.
Additional hidden-state probe controls	Supplementary frozen representation analyses	Within-checkpoint probe evaluation unless explicitly reported across seeds	Representation diagnostic; not used as an independent training-replication claim.
Distributed-history provenance	Persistent and latest-write-only, seeds 42/43/44	Training seed; Student- $t$ ( $df=2$ )	Replicated descriptive direct-provenance result.

Appendix

Table 22: Scope of reported analyses. Training seed is used as the replication unit only where three independently trained checkpoints are available. Sequence-bootstrap intervals characterize within-checkpoint evaluation variation and are not substitutes for training replication.

Explore similar work

Jun 12, 2026cs.LG

Sticky Routing: Training MoE Models for Memory-Efficient Inference

Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping between slow storage and fast memory on edge devices. Existing remedies are either system-level (caching heuristics) or post-hoc (router fine-tuning), leaving the root cause unchanged during pretraining. We propose StickyMoE, a differentiable routing consistency loss that penalises abrupt expert switches between adjacent tokens, encouraging the router to maintain the same expert assignment across semantically coherent spans. StickyMoE requires no architectural changes, adds a single hyperparameter lambda, and unlike post-hoc methods, allows expert representations and routing decisions to co-adapt from the first training step. Experiments on small-scale MoE language models show that StickyMoE reduces the expert switch rate by up to 60% with less than 4% perplexity degradation, Pareto-dominating post-hoc fine-tuning on the quality-locality frontier. Routing temporal locality is most efficiently instilled at training time.

Ali Kayyam

BrainChip Inc., Laguna Hills, CA

Mixture-Of-Experts Large Language Models Mixture-Of-Experts Large Language Model Routing

Ask PDF

Apr 1, 2026cs.AI

Self-Routing: Parameter-Free Expert Routing from Hidden States

Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token, and typically rely on a learned router to map hidden states to expert assignments. In this work, we ask whether a dedicated learned router is strictly necessary for MoE routing. We propose Self-Routing, a parameter-free routing mechanism that uses a designated subspace of the token hidden state directly as expert logits, eliminating the router projection entirely while leaving the rest of the MoE layer unchanged. We evaluate Self-Routing on language modeling across different expert counts and model scales, and on ImageNet-1K classification by comparing it against a standard learned router, random-routing baselines, and dense non-MoE baselines. Our results show that Self-Routing remains competitive with the learned-router baseline while removing all dedicated routing parameters, and yields more balanced expert utilization, with about 17 % higher average normalized routing entropy and no explicit load-balancing loss. On ImageNet-1K with DeiT-S/16, Self-Routing also slightly improves over the corresponding learned-router MoE. These findings suggest that effective MoE routing can emerge from the hidden representation itself without requiring a separate learned router module.

Jama Hussein Mohamud, Drew Wagner, Mirco Ravanelli

Mila - Quebec AI Institute · Universite de Montreal · Concordia University

Mixture-Of-Experts Large Language Model Routing Hidden States

Ask PDF

Aug 9, 2026cs.LG

Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts

Sparse Mixture-of-Experts (MoE) routers commonly use the same scores both to select experts and to weight their already-computed outputs. We study whether these two roles, dispatch and aggregation, should be coupled. On pretrained OLMoE-1B-7B, we keep selected Top-8 expert IDs, expert computation, and total selected router mass fixed and change only within-set aggregation. A structured oracle improves full-horizon cross-entropy by 0.0160 +/- 0.0039 across three seeds; the router's top-scored expert is the counterfactual-best vertex only 17.2% of the time, with router-utility Spearman 0.030. We therefore train Fixed-Dispatch Adaptive Aggregation (FDAA), a 301K-parameter post-compute head optimized directly with the language-modeling objective while freezing the backbone, router, and experts. On OLMoE, FDAA improves fresh WikiText-103 test by Delta CE = -0.1523 +/- 0.0031 across three seeds, and mixed-domain training gives robust gains on WikiText-103, C4, and held-out Penn Treebank under frozen confirmatory evaluation. We also replicate the fixed-dispatch audit on DeepSeek-V2-Lite, which uses Top-6 routed experts plus shared experts. Best-vertex headroom remains significant on WikiText and C4, while router Top1 identifies the best selected expert in only 12.5% and 16.7% of audited examples. In a one-seed mixed-domain replication, FDAA improves locked WikiText and PTB, while C4 is statistically neutral. These results support a cross-architecture distinction between expert selection and expert commitment.

Zongfei Li

Experts Expert Demonstrations Intrusion Detection

Ask PDF