Fewer Expert Transfers Without Changing Routing: Expert Eviction for MoE Offloading
Organizations: CarbonForge, Mila · Université de Montréal · Mila · CarbonForge
Abstract
A mixture-of-experts (MoE) language model uses only a few experts per token, so a server can keep most expert weights in CPU memory and copy an expert to the GPU when the router requests it. Recent methods train the router to keep the same experts for several tokens, and treat fewer routing switches as a proxy for fewer copies. We measure the trade-off directly, in a harness that performs and times real CPU-to-GPU copies for OLMoE-1B-7B. On an L40S GPU, every tested policy that holds an expert set fixed and decodes faster than the least-recently-used (LRU) cache increases next-token loss by at least 0.067 nats. A change that leaves every routing decision intact does better: when the cache must evict an expert, evict the one whose next use lies farthest in the future. This rule copies 2.1x fewer experts than LRU at no loss in quality and is within 3.5% of the fewest copies possible. It needs future requests; the next sixteen tokens of demand recover 96% of its reduction, whereas a rule based on counts of past requests recovers 27%.
Figures & tables
| quality | serving cost | ||
|---|---|---|---|
| Policy | NLL | tok/s | |
| All 64 experts cached | |||
| LRU (lossless) | |||
| Farthest-future eviction | |||
| Dwell-16 (window ) | |||
| Dwell-64 (window ) | |||
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Policy | Accuracy | Retention |
|---|---|---|
| Free routing | ||
| Fixed set, oracle selection | ||
| Fixed set, chosen from recent use | ||
| Learned , seed 0 | ||
| Learned , seed 1 | ||
| Learned , seed 2 |
| Objective | Base NLL | LRU | LRU LFU eviction | Trained layer ( ) |
|---|---|---|---|---|
| Base (untuned) | ||||
| SFT control | — | |||
| Cache-hit reward | ||||
| cache-size tag | — |
Mixture-of-Experts Inference
Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.
See all 200 papers