Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study

Shriniwas Ramesh Suram

cs.AIAug 18, 2026

Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study

Authors: Shriniwas Ramesh Suram

Organizations: University of Cumberlands, Williamsburg, USA

Abstract

Serving a 235B-parameter Mixture-of-Experts (MoE) model on a single 8 GB GPU is bottlenecked not by compute but by memory bandwidth: decode must stream each token's active experts from whichever tier holds them, and on consumer hardware most experts sit on an SSD far slower than RAM. We quantify this bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB): measured decode is 0.44 tok/s warm, matching a bytes-per-token / bandwidth model, while a batching scheme that should amortize one disk sweep instead collapses at batch 32 from paging thrash. We build llama-moe-trace, a zero-surgery router-telemetry tool, and measure routing on Qwen3-30B: adjacent-token expert reuse is 2.0x chance, 95% of traffic uses 52.5% of experts, and an LRU cache of 13.4% of experts serves 66% of requests. We then ask whether cacheability is trainable: we pre-register training of 137M MoE language models with auxiliary locality and domain router losses, under joint criteria on cache-miss reduction and perplexity. The mechanism works (misses down up to 60%; a 99% static-pin hit rate) but every configuration fails the pre-registered <=1% perplexity gate -- miss reduction and quality are tightly coupled. Concurrent StickyMoE reports the same loss as near-free on single-domain sub-25M models; on multi-domain 137M we find the tax real. Our contribution is this pre-registered, stricter-criterion, multi-domain evaluation plus edge-serving measurements. A 340M rung shows the tax does not shrink with scale (it rises slightly). We further show training-free cache-aware rerouting stacks with trained locality -- together ~80% miss reduction at <=3.4% perplexity at both sizes, far cheaper than either alone -- while domain-primed prefetching does not help. All code, traces, and the pre-registration are released.

Figures & tables

Tier	Bandwidth	Capacity	Share of 134 GB model
VRAM (GDDR6)	$\sim$ 448 GB/s	8 GB	6%
System RAM (DDR4)	$\sim$ 50 GB/s	32 GB	24%
NVMe SSD (PCIe 3, DRAM-less)	$\sim$ 2.4 GB/s	$\gg$	$\sim$ 70%

Table 1: Memory hierarchy on the test machine (i9-12900, RTX 3070 8 GB, 32 GB DDR4, WD SN530 NVMe). The 4-bit 235B model is 134 GB; only

\sim

40 GB fits the fast tiers.

Panel of Figure 1 — Figure 1: An MoE layer. The router activates $k{=}2$ of $E{=}128$ experts per token (Qwen3 uses top-8). Idle experts (grey) cost storage but not per-token bandwidth—the sparsity that makes edge serving conceivable.

Condition	Measured	Model
Decode, single stream (warm)	0.441 tok/s	—
Decode, single stream (cold)	0.128 tok/s	0.20 tok/s
Prefill	0.25 tok/s	—
Aggregate, $B{=}1$	0.128 tok/s	0.112
Aggregate, $B{=}8$	0.189 tok/s	0.20
Aggregate, $B{=}32$	0.087 tok/s	0.35

Table 2: Serving measurements, Qwen3-235B-A22B Q4_K_M. Aggregate throughput at batch

B

across concurrent streams. The union-of-experts model predicts aggregate rate should rise with

B

; it does to

B{=}8

then collapses.

Panel of Figure 2 — Figure 2: The bandwidth wall (left) and the batching collapse (right).

Metric	Value
$P(\text{expert reused at next token})$	0.444 (chance 0.223, $2.0\times$ )
Working set (95% of traffic)	52.5% of experts
LRU hit rate @ 13.4% budget	65.9%
LFU hit rate @ 13.4% budget	60.1%
Static-pin hit rate @ 13.4% budget	59.2%
Belady oracle @ 13.4% budget	79.1%

Table 3: Routing profile of Qwen3-30B-A3B ( llama-moe-trace , 8k tokens

\times

4 domains). Cache hit rates at the 235B’s 13.4% fast-memory budget (layer 24, code domain).

	code	general	math	medical
code	1.00	0.16	0.14	0.11
general	0.16	1.00	0.33	0.42
math	0.14	0.33	1.00	0.35
medical	0.11	0.42	0.35	1.00

Table 4: Cross-domain expert-usage similarity (Qwen3-30B-A3B). Code’s expert set is nearly disjoint from the others.

Panel of Figure 3 — Figure 3: Expert-cache hit rate vs. fast-memory budget (Qwen3-30B, layer 24, code). At the 235B’s 13.4% budget (dashed), LRU already achieves 66% and beats static pinning—the signal is temporal, favouring a dynamic resident set.

Panel of Figure 4 — Figure 4: The sticky-moe objective. Arm A uses only $\mathcal{L}_{\text{balance}}$ ; arm B adds $\lambda\mathcal{L}_{\text{loc}}$ ; arm C adds $\mu\mathcal{L}_{\text{dom}}$ . The claim under test is the box on the right at no perplexity cost .

run	arm	$\lambda$	PPL	reuse	LRU hit@25%	static hit@50%
a-main (s1)	A	—	32.0	0.336	0.494	0.747
a-main (s2)	A	—	31.9	0.312	0.480	0.734
b-l02	B	0.02	32.0	0.407	0.564	0.794
b-l03	B	0.03	32.5	0.451	0.608	0.817
b-main (s1)	B	0.05	32.6	0.634	0.788	0.930
b-main (s2)	B	0.05	32.9	0.635	0.796	0.933

Table 5: Main results (200M tokens/arm). Metrics averaged over prose/code/math. “hit” is the fraction of expert-loads already resident.

Panel of Figure 5 — Figure 5: Dose-response of the locality loss (200M tokens). Miss reduction (left, blue) and perplexity cost (right, red) rise together. The RQ1 pass region needs miss reduction above 30% and PPL cost below 1%—the curves never satisfy both simultaneously. (Two axes shown only to display the coupling; they are not a dual-scale claim about one series.)

Panel of Figure 6 — Figure 6: Complementarity of trained and inference-time locality. Training-free rerouting on the baseline (blue) is cheap only up to $\sim$ 30% miss reduction, then diverges. On the locality-trained model the same rerouting (green) stays flat and low, reaching 80% miss reduction at $+$ 2.4% perplexity—far beyond training-time alone (squares, max 59%) or training-free alone. The experts’ co-adaptation makes inference-time substitution nearly free.

	locality tax ( $\tau{=}0$ )		stacked ( $\tau{=}0.5$ )
size	$\Delta$ PPL	miss red.	$\Delta$ PPL	miss red.
137M	$+2.0\%$	$59\%$	$+2.4\%$	$80\%$
340M	$+2.5\%$	$57\%$	$+3.4\%$	$82\%$

Table 6: Scale rung. Locality tax (arm A baseline vs arm B

\lambda{=}0.05

\tau{=}0

) and the stacked training-free rerouting point (

\tau{=}0.5

on arm B), at 137M and 340M; same 200M-token budget, cap

{=}25\%

experts, LRU, seed 1. The tax does not shrink with scale; the stacking advantage persists.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

arm	domain	PPL	$\Delta$ PPL	reuse	hit@25%	static@50%
A ( $\lambda{=}0$ )	prose	63.9	—	0.285	0.432	0.765
A ( $\lambda{=}0$ )	code	4.7	—	0.384	0.544	0.722
A ( $\lambda{=}0$ )	math	27.3	—	0.303	0.486	0.738
B ( $\lambda{=}0.05$ )	prose	65.8	$+$ 3.0%	0.619	0.777	0.930
B ( $\lambda{=}0.05$ )	code	4.7	$+$ 0.0%	0.685	0.829	0.954
B ( $\lambda{=}0.05$ )	math	27.8	$+$ 1.8%	0.599	0.771	0.912

Appendix

Table 7: Per-domain metrics, baseline (A) vs. locality (

\lambda{=}0.05

), each averaged over 2 seeds. “hit@25%” is LRU; “static@50%” is static-pin.

run	arm	$\lambda$	tokens	PPL	reuse	LRU hit@25%	static@50%
a-sanity	A	—	50M	69.3	0.307	0.469	0.731
b-l05	B	0.05	50M	69.5	0.644	0.809	0.957
b-l01	B	0.01	50M	67.7	0.354	0.518	0.779
b-l20	B	0.20	50M	70.1	0.861	0.855	0.977
a-main s1	A	—	200M	32.0	0.336	0.494	0.747
a-main s2	A	—	200M	31.9	0.312	0.480	0.734

Appendix

Table 8: Complete run log. Metrics averaged over prose/code/math.

Model		Optimization
$d_{\text{model}}$	384	optimizer	AdamW ( $\beta{=}0.9,0.95$ )
layers	8	weight decay	0.1
attention heads	6	peak LR	$6\times 10^{-4}$
experts $E$	16	schedule	one-cycle, 2% warmup
top- $k$	2	grad clip	1.0
expert $d_{\text{ff}}$	768	precision	bf16 autocast

Appendix

Table 9: Hyperparameters (identical across arms; only the router-loss weights

\lambda,\mu

differ).

Mixture-of-Experts Language Models

Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.

See all 225 papers

Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference
Sep 4, 2026Zhenhe Wu, Yaping Jin, Qinghua Xing +6Mixture of Experts Memory-Efficient Optimization
Huawei Technologies · Beihang University · Tianjin University +2
Sticky Routing: Training MoE Models for Memory-Efficient Inference
Jun 12, 2026Ali KayyamMixture-of-Experts Inference Memory-Efficient Inference
BrainChip Inc., Laguna Hills, CA
MoRE: Scaling mixture of experts with hardware-aware low-rank routing
Sep 28, 2026Honam Wong, Surbhi Goel, Enric Boix-AdseràParameter-Free Mixture-of-Experts Routing Low-Rank Matrix Decomposition
University of Pennsylvania · The Wharton School, University of Pennsylvania

Sep 4, 2026J/K move · Enter open · S save

Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference

Zhenhe Wu, Yaping Jin, Qinghua Xing, Hang Zhou+5

Huawei Technologies · Beihang University · Tianjin University+2

Mixture of Experts Memory-Efficient Optimization Mixture-of-Experts Inference Memory-Efficient Inference Expert Routing

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models activate few experts per token, yet their full expert sets can exceed GPU memory and require repeated weight transfers during decoding. We formulate expert-cache management as a model-side algorithmic problem and propose cache-aware post-training that jointly adapts the MoE backbone and lightweight auxiliary routers while preserving the native inference-time Top-K rule. The update-only Temporal Router learns same-layer retention across tokens without proactive loading. The full Spatio-Temporal Router adds a Spatio Router that uses the causal predecessor's hidden state to refine the temporal cache before target-layer access. We evaluate both modes on Qwen3 and GPT-OSS across GSM8K, MATH, and CommonsenseQA. Temporal Router consistently improves hit rate and reduces expert-weight traffic over matched LM-only baselines. On Qwen3, the full mode improves adjusted hit rate by 1.15--18.03 points and reduces traffic by 4.6--53.3% relative to the strongest evaluated prefetching baseline; GPT-OSS results are competitive but task-dependent. Auxiliary-only training preserves baseline accuracy but yields modest coverage gains; joint post-training achieves substantially higher coverage. Sensitivity analyses distinguish the effects of cache capacity, refinement budget, and cache-loss weight on coverage, traffic, and quality.

Abstract

Mixture-of-Experts Language Models

Explore similar work

Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference

Sticky Routing: Training MoE Models for Memory-Efficient Inference

MoRE: Scaling mixture of experts with hardware-aware low-rank routing