Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study

Shriniwas Ramesh Suram

cs.AIAug 18, 2026

Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study

Authors: Shriniwas Ramesh Suram

Organizations: University of Cumberlands, Williamsburg, USA

Abstract

Serving a 235B-parameter Mixture-of-Experts (MoE) model on a single 8 GB GPU is bottlenecked not by compute but by memory bandwidth: decode must stream each token's active experts from whichever tier holds them, and on consumer hardware most experts sit on an SSD far slower than RAM. We quantify this bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB): measured decode is 0.44 tok/s warm, matching a bytes-per-token / bandwidth model, while a batching scheme that should amortize one disk sweep instead collapses at batch 32 from paging thrash. We build llama-moe-trace, a zero-surgery router-telemetry tool, and measure routing on Qwen3-30B: adjacent-token expert reuse is 2.0x chance, 95% of traffic uses 52.5% of experts, and an LRU cache of 13.4% of experts serves 66% of requests. We then ask whether cacheability is trainable: we pre-register training of 137M MoE language models with auxiliary locality and domain router losses, under joint criteria on cache-miss reduction and perplexity. The mechanism works (misses down up to 60%; a 99% static-pin hit rate) but every configuration fails the pre-registered <=1% perplexity gate -- miss reduction and quality are tightly coupled. Concurrent StickyMoE reports the same loss as near-free on single-domain sub-25M models; on multi-domain 137M we find the tax real. Our contribution is this pre-registered, stricter-criterion, multi-domain evaluation plus edge-serving measurements. A 340M rung shows the tax does not shrink with scale (it rises slightly). We further show training-free cache-aware rerouting stacks with trained locality -- together ~80% miss reduction at <=3.4% perplexity at both sizes, far cheaper than either alone -- while domain-primed prefetching does not help. All code, traces, and the pre-registration are released.

Figures & tables

Tier	Bandwidth	Capacity	Share of 134 GB model
VRAM (GDDR6)	$\sim$ 448 GB/s	8 GB	6%
System RAM (DDR4)	$\sim$ 50 GB/s	32 GB	24%
NVMe SSD (PCIe 3, DRAM-less)	$\sim$ 2.4 GB/s	$\gg$	$\sim$ 70%

Table 1: Memory hierarchy on the test machine (i9-12900, RTX 3070 8 GB, 32 GB DDR4, WD SN530 NVMe). The 4-bit 235B model is 134 GB; only

\sim

40 GB fits the fast tiers.

Panel of Figure 1 — Figure 1: An MoE layer. The router activates $k{=}2$ of $E{=}128$ experts per token (Qwen3 uses top-8). Idle experts (grey) cost storage but not per-token bandwidth—the sparsity that makes edge serving conceivable.

Condition	Measured	Model
Decode, single stream (warm)	0.441 tok/s	—
Decode, single stream (cold)	0.128 tok/s	0.20 tok/s
Prefill	0.25 tok/s	—
Aggregate, $B{=}1$	0.128 tok/s	0.112
Aggregate, $B{=}8$	0.189 tok/s	0.20
Aggregate, $B{=}32$	0.087 tok/s	0.35

Table 2: Serving measurements, Qwen3-235B-A22B Q4_K_M. Aggregate throughput at batch

B

across concurrent streams. The union-of-experts model predicts aggregate rate should rise with

B

; it does to

B{=}8

then collapses.

Panel of Figure 2 — Figure 2: The bandwidth wall (left) and the batching collapse (right).

Metric	Value
$P(\text{expert reused at next token})$	0.444 (chance 0.223, $2.0\times$ )
Working set (95% of traffic)	52.5% of experts
LRU hit rate @ 13.4% budget	65.9%
LFU hit rate @ 13.4% budget	60.1%
Static-pin hit rate @ 13.4% budget	59.2%
Belady oracle @ 13.4% budget	79.1%

Table 3: Routing profile of Qwen3-30B-A3B ( llama-moe-trace , 8k tokens

\times

4 domains). Cache hit rates at the 235B’s 13.4% fast-memory budget (layer 24, code domain).

	code	general	math	medical
code	1.00	0.16	0.14	0.11
general	0.16	1.00	0.33	0.42
math	0.14	0.33	1.00	0.35
medical	0.11	0.42	0.35	1.00

Table 4: Cross-domain expert-usage similarity (Qwen3-30B-A3B). Code’s expert set is nearly disjoint from the others.

Panel of Figure 3 — Figure 3: Expert-cache hit rate vs. fast-memory budget (Qwen3-30B, layer 24, code). At the 235B’s 13.4% budget (dashed), LRU already achieves 66% and beats static pinning—the signal is temporal, favouring a dynamic resident set.

Panel of Figure 4 — Figure 4: The sticky-moe objective. Arm A uses only $\mathcal{L}_{\text{balance}}$ ; arm B adds $\lambda\mathcal{L}_{\text{loc}}$ ; arm C adds $\mu\mathcal{L}_{\text{dom}}$ . The claim under test is the box on the right at no perplexity cost .

run	arm	$\lambda$	PPL	reuse	LRU hit@25%	static hit@50%
a-main (s1)	A	—	32.0	0.336	0.494	0.747
a-main (s2)	A	—	31.9	0.312	0.480	0.734
b-l02	B	0.02	32.0	0.407	0.564	0.794
b-l03	B	0.03	32.5	0.451	0.608	0.817
b-main (s1)	B	0.05	32.6	0.634	0.788	0.930
b-main (s2)	B	0.05	32.9	0.635	0.796	0.933

Table 5: Main results (200M tokens/arm). Metrics averaged over prose/code/math. “hit” is the fraction of expert-loads already resident.

Panel of Figure 5 — Figure 5: Dose-response of the locality loss (200M tokens). Miss reduction (left, blue) and perplexity cost (right, red) rise together. The RQ1 pass region needs miss reduction above 30% and PPL cost below 1%—the curves never satisfy both simultaneously. (Two axes shown only to display the coupling; they are not a dual-scale claim about one series.)

Panel of Figure 6 — Figure 6: Complementarity of trained and inference-time locality. Training-free rerouting on the baseline (blue) is cheap only up to $\sim$ 30% miss reduction, then diverges. On the locality-trained model the same rerouting (green) stays flat and low, reaching 80% miss reduction at $+$ 2.4% perplexity—far beyond training-time alone (squares, max 59%) or training-free alone. The experts’ co-adaptation makes inference-time substitution nearly free.

	locality tax ( $\tau{=}0$ )		stacked ( $\tau{=}0.5$ )
size	$\Delta$ PPL	miss red.	$\Delta$ PPL	miss red.
137M	$+2.0\%$	$59\%$	$+2.4\%$	$80\%$
340M	$+2.5\%$	$57\%$	$+3.4\%$	$82\%$

Table 6: Scale rung. Locality tax (arm A baseline vs arm B

\lambda{=}0.05

\tau{=}0

) and the stacked training-free rerouting point (

\tau{=}0.5

on arm B), at 137M and 340M; same 200M-token budget, cap

{=}25\%

experts, LRU, seed 1. The tax does not shrink with scale; the stacking advantage persists.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

arm	domain	PPL	$\Delta$ PPL	reuse	hit@25%	static@50%
A ( $\lambda{=}0$ )	prose	63.9	—	0.285	0.432	0.765
A ( $\lambda{=}0$ )	code	4.7	—	0.384	0.544	0.722
A ( $\lambda{=}0$ )	math	27.3	—	0.303	0.486	0.738
B ( $\lambda{=}0.05$ )	prose	65.8	$+$ 3.0%	0.619	0.777	0.930
B ( $\lambda{=}0.05$ )	code	4.7	$+$ 0.0%	0.685	0.829	0.954
B ( $\lambda{=}0.05$ )	math	27.8	$+$ 1.8%	0.599	0.771	0.912

Appendix

Table 7: Per-domain metrics, baseline (A) vs. locality (

\lambda{=}0.05

), each averaged over 2 seeds. “hit@25%” is LRU; “static@50%” is static-pin.

run	arm	$\lambda$	tokens	PPL	reuse	LRU hit@25%	static@50%
a-sanity	A	—	50M	69.3	0.307	0.469	0.731
b-l05	B	0.05	50M	69.5	0.644	0.809	0.957
b-l01	B	0.01	50M	67.7	0.354	0.518	0.779
b-l20	B	0.20	50M	70.1	0.861	0.855	0.977
a-main s1	A	—	200M	32.0	0.336	0.494	0.747
a-main s2	A	—	200M	31.9	0.312	0.480	0.734

Appendix

Table 8: Complete run log. Metrics averaged over prose/code/math.

Model		Optimization
$d_{\text{model}}$	384	optimizer	AdamW ( $\beta{=}0.9,0.95$ )
layers	8	weight decay	0.1
attention heads	6	peak LR	$6\times 10^{-4}$
experts $E$	16	schedule	one-cycle, 2% warmup
top- $k$	2	grad clip	1.0
expert $d_{\text{ff}}$	768	precision	bf16 autocast

Appendix

Table 9: Hyperparameters (identical across arms; only the router-loss weights

\lambda,\mu

differ).

Mixture-of-Experts Language Models

Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.

See all 225 papers

Sep 4, 2026

Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference

Mixture-of-Experts (MoE) models activate few experts per token, yet their full expert sets can exceed GPU memory and require repeated weight transfers during decoding. We formulate expert-cache management as a model-side algorithmic problem and propose cache-aware post-training that jointly adapts the MoE backbone and lightweight auxiliary routers while preserving the native inference-time Top-K rule. The update-only Temporal Router learns same-layer retention across tokens without proactive loading. The full Spatio-Temporal Router adds a Spatio Router that uses the causal predecessor's hidden state to refine the temporal cache before target-layer access. We evaluate both modes on Qwen3 and GPT-OSS across GSM8K, MATH, and CommonsenseQA. Temporal Router consistently improves hit rate and reduces expert-weight traffic over matched LM-only baselines. On Qwen3, the full mode improves adjusted hit rate by 1.15--18.03 points and reduces traffic by 4.6--53.3% relative to the strongest evaluated prefetching baseline; GPT-OSS results are competitive but task-dependent. Auxiliary-only training preserves baseline accuracy but yields modest coverage gains; joint post-training achieves substantially higher coverage. Sensitivity analyses distinguish the effects of cache capacity, refinement budget, and cache-loss weight on coverage, traffic, and quality.

Zhenhe Wu, Yaping Jin, Qinghua Xing +6

Huawei Technologies · Beihang University · Tianjin University +2

Mixture of Experts Memory-Efficient Optimization Mixture-of-Experts Inference Memory-Efficient Inference Expert Routing

Ask PDF

Jun 12, 2026

Sticky Routing: Training MoE Models for Memory-Efficient Inference

Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping between slow storage and fast memory on edge devices. Existing remedies are either system-level (caching heuristics) or post-hoc (router fine-tuning), leaving the root cause unchanged during pretraining. We propose StickyMoE, a differentiable routing consistency loss that penalises abrupt expert switches between adjacent tokens, encouraging the router to maintain the same expert assignment across semantically coherent spans. StickyMoE requires no architectural changes, adds a single hyperparameter lambda, and unlike post-hoc methods, allows expert representations and routing decisions to co-adapt from the first training step. Experiments on small-scale MoE language models show that StickyMoE reduces the expert switch rate by up to 60% with less than 4% perplexity degradation, Pareto-dominating post-hoc fine-tuning on the quality-locality frontier. Routing temporal locality is most efficiently instilled at training time.

Ali Kayyam

BrainChip Inc., Laguna Hills, CA

Mixture-of-Experts Inference Memory-Efficient Inference Expert Routing Mixture-of-Experts Models

Ask PDF

Sep 28, 2026

MoRE: Scaling mixture of experts with hardware-aware low-rank routing

Mixture-of-Experts (MoE) layers are central to frontier language models, and recent architectures push toward more and smaller experts. In this regime, the standard linear router becomes a bottleneck: with

M

experts and hidden dimension

h

, its per-token cost

Θ(Mh)

dominates the MoE layer once

M

is large. We introduce MoRE (Mixture of Rank-reduced-routed Experts), which factorizes the router weight matrix at rank

r

and reduces the routing cost to

O((h + M)r)

. We prove that rank logarithmic in

M

suffices for routing expressivity when the number of active experts is fixed, and is necessary up to precision factors. We also prove that logarithmic rank preserves load balance in a Gaussian memorization model, and training on a synthetic phonebook task shows that low rank does not hurt memorization. At matched active FLOPs, the factorization allows a factor of

Θ(h/r)

more experts. To realize this gain in wall-clock time, we design a fused Triton kernel at inference that avoids expensive memory operations on HBM. Empirically, MoRE improves memorization on the phonebook task and performance on knowledge-intensive Q&A benchmarks after pretraining, while matching reasoning ability. Code available at https://github.com/Matheart/MoRE_code.

Honam Wong, Surbhi Goel, Enric Boix-Adserà

University of Pennsylvania · The Wharton School, University of Pennsylvania

Parameter-Free Mixture-of-Experts Routing Low-Rank Matrix Decomposition Mixture-of-Experts Inference LLM Inference Acceleration Mixture-of-Experts Models

Ask PDF

Abstract

Mixture-of-Experts Language Models

Explore similar work

Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference

Sticky Routing: Training MoE Models for Memory-Efficient Inference

MoRE: Scaling mixture of experts with hardware-aware low-rank routing