A mixture-of-experts (MoE) language model uses only a few experts per token, so a server can keep most expert weights in CPU memory and copy an expert to the GPU when the router requests it. Recent methods train the router to keep the same experts for several tokens, and treat fewer routing switches as a proxy for fewer copies. We measure the trade-off directly, in a harness that performs and times real CPU-to-GPU copies for OLMoE-1B-7B. On an L40S GPU, every tested policy that holds an expert set fixed and decodes faster than the least-recently-used (LRU) cache increases next-token loss by at least 0.067 nats. A change that leaves every routing decision intact does better: when the cache must evict an expert, evict the one whose next use lies farthest in the future. This rule copies 2.1x fewer experts than LRU at no loss in quality and is within 3.5% of the fewest copies possible. It needs future requests; the next sixteen tokens of demand recover 96% of its reduction, whereas a rule based on counts of past requests recovers 27%.
Figures & tables
Figure 1: Quality loss against decoding speed as expert copies become more expensive. Left: measured L40S throughput for policies run in the harness; the farthest-future points combine their counted copies with the measured compute time. Middle and right: Equation ( 1 ) with copies 10× and 50× slower; among six policies checked by direct measurement, it picks the measured winner at both costs (§ 3.1 ). Points at Δ NLL =0 preserve every routing decision. At the native cost, no policy that holds an expert set fixed decodes faster than LRU without at least 0.067 nats of extra loss, while farthest-future eviction loses nothing and copies 2.1× fewer experts.
quality
serving cost
Policy
Δ NLL
tok/s
ρ
All 64 experts cached
+0.000
49.0
0.0
LRU (lossless)
+0.000
38.3
9.6
Farthest-future eviction
+0.000
42.5
4.6
Dwell-16 (window 8× )
+0.067
42.2
4.5
Dwell-64 (window 1× )
+0.085
43.5
3.9
Table 1: Left: quality, speed and steady-state copies per token ( ρ ) on the L40S, where one expert copy takes τ=0.51 ms. The farthest-future throughput is computed from Equation ( 1 ); the others are measured. Shaded policies preserve every routing decision and match the NLL of the all-64-expert baseline. Right: copy time and bandwidth across GPUs; raw logs were kept only for the L40S.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Share of the gap between LRU and the oracle that the oracle recovers when it sees only the next k tokens of requests. The dotted lines are the causal rules, which correspond to about two tokens of foresight.
Policy
Accuracy
Retention
Free routing
0.484
100.0%
Fixed set, oracle selection
0.481
99.5%
Fixed set, chosen from recent use
0.430
88.9%
Learned βμ , seed 0
0.422
87.3%
Learned βμ , seed 1
0.415
85.7%
Learned βμ , seed 2
0.401
82.8%
Appendix
Table 2: MMLU accuracy at a matched 3.1% switch rate ( n=5,000 ; aggregate summaries). The ±1.3 is one standard error across three training seeds.
Objective
Base NLL
LRU
LRU + LFU eviction
Trained layer ( C=16 )
Base (untuned)
1.382
31,915
27,275
10,954
SFT control
1.350
31,686
27,107
—
Cache-hit reward
1.397
32,050
27,355
10,963
+ cache-size tag
—
33,101
29,063
11,223
Appendix
Table 3: Steady expert transfers (lower is better) for routers fine-tuned for cache hits, under lossless LRU serving, under lossless serving with decayed-frequency (LFU) eviction, and at the method’s training setting.
Mixture-of-Experts (MoE) language models route each token to a small subset of experts, but whether the routes selected by a trained top-k router are good ones is rarely evaluated directly. Holding the model fixed, we compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory. The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning, where lower-loss equal-compute routes consistently exist inside the frozen model but are not selected. The same pattern holds across Qwen3-30B-A3B, GPT-OSS-20B, DeepSeek-V2-Lite, and OLMoE-1B-7B, and follows structurally from how standard top-k training evaluates routing decisions: the language modeling loss scores only the executed route, and load balancing depends only on aggregate routing statistics. A minimal router-only update to the final-layer router, leaving every expert and every other router frozen, is sufficient to shift pass@K on AIME 2024+2025 and HMMT 2025 for both Qwen3-30B-A3B and GPT-OSS-20B, suggesting that at least part of the failure reflects router-reachable misallocation rather than expert capacity alone.
Youngsik Yoon, Siwei Wang, Wei Chen +1
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea
Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping between slow storage and fast memory on edge devices. Existing remedies are either system-level (caching heuristics) or post-hoc (router fine-tuning), leaving the root cause unchanged during pretraining. We propose StickyMoE, a differentiable routing consistency loss that penalises abrupt expert switches between adjacent tokens, encouraging the router to maintain the same expert assignment across semantically coherent spans. StickyMoE requires no architectural changes, adds a single hyperparameter lambda, and unlike post-hoc methods, allows expert representations and routing decisions to co-adapt from the first training step. Experiments on small-scale MoE language models show that StickyMoE reduces the expert switch rate by up to 60% with less than 4% perplexity degradation, Pareto-dominating post-hoc fine-tuning on the quality-locality frontier. Routing temporal locality is most efficiently instilled at training time.
Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.