cs.AIOct 26, 2026

Prefetch-Steered Routing: Deterministic Cache-Aware Expert Selection for Offloaded Mixture-of-Experts Decoding

Authors: Siming Huang

Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

When a Mixture-of-Experts (MoE) model does not fit in GPU memory, batch-1 decoding is dominated by copying expert weights from host memory. Cache-aware routing reduces these copies by biasing the router toward resident experts, but combined with asynchronous prefetching it makes the model's outputs depend on transfer timing, so a harness that does not simulate that timing no longer evaluates the deployed model. We propose prefetch-steered routing (PSR): routing conditions on a deterministic logical cache, into which at most n experts predicted by the next layer's router are inserted per layer and token; physical copies affect only latency. On OLMoE-1B-7B with 50% and 25% of experts resident on an RTX 4090, PSR decodes 2.2-2.3x faster than exact routing with LRU and 25-30% faster than the Cache-Prior operating point, with mean benchmark differences of +0.1 and +0.5 points (95% lower bounds -0.8 and -0.3). In greedy generation its lead over Cache-Prior depends on the workload and reverses on HumanEval at 50% residency. On Moonlight-16B-A3B, PSR is 1.9x faster than exact routing with LRU at 37.5% residency and, at the latency of Cache-Prior, significantly better on three of eight benchmarks. On Qwen3.6-35B-A3B, where copies are cheap relative to computation, PSR keeps its advantage over Cache-Prior on the likelihood benchmarks but not its speed. Distillation-trained gates did not help. Speculative decoding with block-level expert selection was slower than PSR with a 1B autoregressive drafter on OLMoE, but with a block drafter on Qwen3.6 it was 2.1-2.9x faster than any autoregressive configuration with at most its GPU memory on the 64 prompts we timed.

Figures & tables

prefetch copies in flight124
C=32C=32 , λ=0.2\lambda=0.2 , cross-layer prefetch of 8
physical cache8.755 (0.041)8.674 (0.054)8.627 (0.048)
logical cache8.509 (0.000)8.509 (0.000)8.509 (0.000)
C=16C=16 , λ=0.25\lambda=0.25 , cross-layer prefetch of 8
physical cache9.097 (0.013)9.015 (0.013)8.864 (0.027)
logical cache8.587 (0.000)8.587 (0.000)8.587 (0.000)
Table 1: Perplexity (12 windows) of Cache-Prior routing with A eviction and cross-layer prefetch of 8 when only the number of prefetch copies in flight changes; mean of 3 runs, with the range across runs in parentheses. Physical cache: routing sees the experts whose copies have been issued (the scheduler also prefetches by token prior). Logical cache: routing sees the logical cache of PSR (unbounded insertion). With routing on the physical cache, outputs depend on the scheduler parameter and vary between runs; with the logical cache they do not.
 C=32C=32 (50%)C=16C=16 (25%)
configsg%msp99copies (stall)sg%msp99copies (stall)
all experts resident (C=64)0.06.870.0 (0.0)(same)
exact routing, LRU0.024.75230.5 (30.5)0.042.56961.4 (61.4)
exact routing, A eviction0.022.94827.5 (27.5)0.041.46859.2 (59.2)
exact routing, best cache + prefetch †0.020.54330.7 (16.9)0.037.86362.5 (42.6)
exact routing, logical prefetch ( n=8n{=}8 )0.020.94431.6 (16.7)0.039.06465.2 (43.8)
Table 2: Engine latency on OLMoE-1B-7B, RTX 4090. sg% = selfgen excess perplexity (KL diagnostic, lower is closer to exact routing). ms = mean per-token latency, p99 = 99th percentile, copies = expert copies per token; in parentheses, demand copies, issued only when the layer needs the expert. Prefetch copies that are still in flight when needed also delay the layer ( Figure 4 ). † Routing or caching depends on transfer timing; for exact routing this affects latency only. PSR rows use A eviction (except the row marked LRU) and cross-layer prefetch with a budget of nn new insertions per layer and token.
Figure 1: Latency vs. divergence from exact routing (selfgen excess, a KL diagnostic) for λ∈{0.1,0.15,0.2,0.25,0.3}\lambda\in\{0.1,0.15,0.2,0.25,0.3\} (labels on Cache-Prior and n=1n{=}1 curves). PSR with a budget of one insertion per layer and token is the fastest configuration at every divergence level its λ\lambda sweep covers; unbounded prefetch is closest to exact but slow. Quality claims rest on the benchmarks of Figure 3 .
Figure 2: Distribution of per-token decode latency at 50% (top) and 25% (bottom) residency: fraction of tokens slower than xx (log scale), over all tokens of the 3 timing repetitions.
CCconfigmsMMLUARC-CHellaS.GSM8KBBHIFE-PIFE-IHumanE.MBPPmean
–exact routing24.7/42.552.449.060.633.236.041.252.332.928.8–
32CP λ\lambda 0.214.3−- 0.3−- 0.5−- 0.9+0.3−- 0.6−- 2.2+0.5+3.0+0.2−- 0.13
32CP λ\lambda 0.311.1−- 1.1−- 2.3 †−- 2.4+0.1−- 2.9 †−- 2.4−- 1.8−- 1.8−- 0.6−- 1.69
32PSR λ\lambda 0.2 nn 112.0−- 0.7 †+0.1−- 0.2+1.3−- 0.2−- 0.7−- 0.4+0.6−- 0.4−- 0.04
32PSR λ\lambda 0.3 nn 110.7−- 0.8 †−- 0.5−- 0.4 †+0.2−- 0.9−- 2.4−- 2.5+5.5 †−- 0.8−- 0.03
32spec. exec.20.0−- 1.3+1.5−- 0.9−- 0.7−- 1.9 †−- 2.0−- 0.2+1.2+1.0−- 0.39
Table 3: Benchmark scores relative to exact routing (points; paired over identical questions; bold : significant after Holm’s correction over the table’s 90 tests; † : p<0.05p<0.05 before correction only). The first row gives absolute scores of exact routing. ms is engine latency at the configuration’s residency. IFE-P/IFE-I = IFEval strict prompt/instruction accuracy. Mean = average over the eight tasks with IFE-P. Speculative execution’s outputs do not depend on residency; its two rows share one benchmark run.
CCPSR (ms)vs. (ms)MMLUARC-CHellaS.GSM8KBBHIFE-PIFE-IHumanE.MBPPmean
iso-latency
32PSR λ\lambda 0.3 nn 1 (10.7)CP λ\lambda 0.3 (11.1)+0.3+1.8+2.0+0.1+2.0 †0.0−- 0.7+7.3 †−- 0.2+1.7 ±\pm 1.0
16PSR λ\lambda 0.25 nn 1 (19.2)CP λ\lambda 0.3 (20.7)+1.2+0.3+1.9+3.4 †+2.3 †+3.5+3.8 †+5.5−- 0.2+2.2 ±\pm 1.0
vs. CP operating point
32PSR λ\lambda 0.3 nn 1 (10.7)CP λ\lambda 0.2 (14.3)−- 0.50.0+0.4 †−- 0.2−- 0.3−- 0.2−- 3.0+2.4−- 1.0+0.1 ±\pm 0.9
32PSR λ\lambda 0.2 nn 1 (12.0)CP λ\lambda 0.2 (14.3)−- 0.4+0.6+0.6+1.0+0.4+1.5−- 0.8−- 2.4−- 0.6+0.1 ±\pm 0.8
Table 4: Direct paired comparisons of PSR with Cache-Prior (and speculative execution) on identical questions (points; bold : significant after Holm’s correction over the table’s 81 tests; † : p<0.05p<0.05 before correction only). Mean ±\pm 95% interval over the eight tasks. Iso-latency pairs run at about the same ms/token; the other pairs compare PSR with the Cache-Prior operating point of Table 3 ( λ=0.2\lambda=0.2 ), which is slower. A mean difference that is not significant does not show equal quality; its 95% lower bound limits how much worse PSR can be on average. Speculative execution does not depend on residency, so one benchmark run serves both CC .
Figure 3: Mean benchmark difference to exact routing over eight tasks ( ±\pm 95% interval) against decode latency, at 50% (top) and 25% (bottom) residency.
taskharnessengineengine −- harness
PSR λ\lambda 0.3 nn 1 vs. CP λ\lambda 0.2
GSM8K−- 0.2−- 0.8−- 0.7 ±\pm 3.5
IFEval-P−- 0.2+1.7+1.8 ±\pm 4.3
IFEval-I−- 3.0+0.7+3.7 † ±\pm 3.5
HumanEval+2.4−- 1.8−- 4.3 ±\pm 4.6
PSR λ\lambda 0.2 nn 1 vs. CP λ\lambda 0.2
Table 5: Engine-generated benchmarks against the harness at 50% residency: paired difference between two configurations (points) in the harness and in the engine on the same questions, and the difference of the two with its 95% interval ( † : p<0.05p<0.05 before Holm’s correction over the table’s 20 tests; none survives it).
 TFGSM8KIFEvalHumanEval
configmsmsp99cp.msp99cp.msp99cp.
50% residency
exact, LRU24.721.644.326.422.451.927.214.334.613.5
CP λ\lambda 0.214.310.823.45.212.129.57.58.114.61.4
PSR λ\lambda 0.3 nn 110.79.514.010.99.918.312.08.411.26.9
PSR λ\lambda 0.2 nn 112.010.219.212.310.722.813.08.613.47.1
Table 6: Latency in greedy free generation on 77 benchmark prompts (decode only; ms per generated token, its 99th percentile, and expert copies per token; means over 3 interleaved rounds), next to the teacher-forced latency (TF) of Table 2 . Ratio rows: PSR over Cache-Prior per task, with the 95% interval over rounds (paired on log latency), which measures run-to-run repeatability on this fixed prompt set; bold: significant after Holm’s correction over the table’s 9 ratios. Over prompts: 95% interval of the same ratio from resampling the task’s prompts, and the number of prompts on which PSR is faster per token.
Figure 4: Expert copies per token at 50% (top) and 25% (bottom) residency: demand copies, issued when the layer needs the expert (full wait); prefetch copies of executed experts that are still in flight when the layer is scheduled (partial wait) or that arrived in time; and prefetch copies whose expert was never executed. Dots: ms/token (right axis).
Figure 5: Moonlight-16B-A3B: latency vs. KL divergence from exact routing at three cache sizes. PSR uses n=1n=1 and a pre-gate window of 4 candidates (lines), 8 or 2 (single points); labels give λ\lambda . Exact routing with logical prefetch is PSR’s prefetching without the bonus. Means over 3 interleaved rounds with 95% intervals. Off exact routing, every point on the latency–KL Pareto front is a PSR configuration.
configmscopiesKLMMLUARC-CHellaS.GSM8KBBHIFE-PIFE-IHumanE.MBPPmean
exact, LRU43.636.70.00067.957.562.274.566.644.256.058.557.8–
exact + prefetch34.641.30.000outputs identical to exact routing
CP-A λ\lambda 0.2 (A eviction)25.913.40.028−- 0.6−- 0.7−- 0.3 †+0.8−- 2.1 †−- 1.7−- 1.8−- 4.3−- 2.2−- 1.37
CP λ\lambda 0.4 (LRU)21.58.00.072−- 1.1−- 2.0 †−- 0.8−- 1.6−- 3.7−- 5.5 †−- 5.9−- 1.2−- 0.2−- 2.03
PSR λ\lambda 0.321.316.80.029−- 0.6 †−- 1.7 †−- 0.4 †+0.6−- 1.7 †−- 0.9−- 1.1−- 2.4−- 1.0−- 1.03
PSR λ\lambda 0.222.618.50.021−- 0.3−- 0.9−- 0.3 †+1.8−- 1.5 †−- 0.9−- 0.6−- 1.2−- 0.6−- 0.48
Table 7: Moonlight-16B-A3B with C=24C=24 of 64 experts resident (37.5%). ms = mean engine latency per token over 3 rounds; copies = expert copies per token; KL from exact routing. Benchmark columns as in Table 3 : the first row gives the absolute scores of exact routing, the other rows the paired difference to it ( bold : significant after Holm’s correction over the table’s 54 tests; † : p<0.05p<0.05 before correction only); the last two rows compare PSR λ=0.2\lambda=0.2 directly with the two Cache-Prior configurations at about the same latency (iso-lat.) and the closest KL (sim. KL), with the mean ±\pm 95% interval. PSR uses the A priority, n=1n=1 and a pre-gate window of 4.
Figure 6: Qwen3.6-35B-A3B (FP8): latency vs. KL divergence from exact routing at three cache sizes of 256 experts per layer. PSR uses n=1n=1 and a pre-gate window of 4 or 2 candidates; labels give λ\lambda . Means over 3 interleaved rounds with 95% intervals. Every cache-aware configuration runs within 3 ms of the fastest; at the fast end of the Pareto front, it includes Cache-Prior configurations.
configmscopiesKLMMLUARC-CHellaS.GSM8KIFE-PIFE-IHumanE.mean
exact, LRU30.053.50.00083.654.663.495.583.588.758.5–
exact + prefetch27.856.20.000outputs identical to exact routing
CP λ\lambda 0.2 (LRU)23.68.30.051−- 1.3−- 1.5−- 0.8−- 0.3−- 0.4+0.1+3.0−- 0.21
PSR λ\lambda 0.224.116.50.029−- 0.4 †−- 0.4−- 0.3−- 0.7−- 2.6−- 1.6+0.6−- 0.63
PSR vs. CP24.1 vs. 23.6 ms+0.8+1.1+0.6−- 0.4−- 2.2−- 1.7−- 2.4−- 0.4 ±\pm 1.0
Table 8: Qwen3.6-35B-A3B (FP8) with C=64C=64 of 256 experts resident (25%). ms = mean engine latency per token over 3 rounds; copies = expert copies per token; KL from exact routing. Benchmark columns as in Table 7 , on a reduced suite of six tasks ( Appendix A ): the first row gives the absolute scores of exact routing, the other rows the paired difference to it ( bold : significant after Holm’s correction over the table’s 21 tests; † : p<0.05p<0.05 before correction only); mean over the six tasks with IFE-P. The last row compares PSR directly with Cache-Prior, with the mean ±\pm 95% interval. Both configurations were fixed before any benchmark was run.
 HumanEvalGSM8K
configpass@1τ\taucopiesmsstrictτ\taucopiesms
AR, all experts resident (exact routing)32.9–0.06.133.2–0.06.5
AR-PSR λ\lambda 0.2530.5–10.09.232.8–15.211.3
AR-PSR λ\lambda 0.334.8–9.89.133.4–14.110.5
SD, exact verification31.16.529.327.333.14.959.753.2
SD, block selection, position weights35.46.76.213.634.3  –
Table 9: Speculative decoding (SD) on OLMoE with P=24P=24 experts per layer (37.5%), RTX 4090, batch 1. pass@1 and strict: full benchmarks (164 and 1319 problems) in the harness. τ\tau = committed tokens per verification, copies = expert copies per generated token, ms = decode latency per generated token: all three from the engines on the timed subset (the first 80 HumanEval and 48 GSM8K problems). The copy ratios quoted in Section 7 use the copies of the full benchmark runs instead, which have longer accepted runs on HumanEval. AR-PSR = PSR with C=24C=24 and n=1n=1 . SD rows use k=16k=16 drafts from OLMo-1B unless noted. Position weights are AcceptMoE’s; confidence + top-1 is our variant ( Section 7 ). “Copies skipped” runs the same routing without any PCIe copy and gives the compute-only time; it and k=8k=8 were only timed. –: not measured.
 GSM8K (32 prompts)HumanEval (32 prompts)
configmscopiesτ\taustrictmscopiesτ\taupass@1
AR C=64C{=}64 , exact routing, LRU36.385.0–78.136.082.6–100.0
AR C=64C{=}64 , CP λ\lambda 0.224.89.2–75.024.36.4–100.0
AR C=64C{=}64 , PSR λ\lambda 0.225.824.7–81.225.723.9–100.0
AR C=128C{=}128 , exact routing, LRU27.928.7–78.128.030.0–100.0
AR C=128C{=}128 , CP λ\lambda 0.224.42.5–84.424.21.8–100.0
Table 10: Qwen3.6-35B-A3B (FP8): speculative decoding (SD) with the DFlash block drafter against AR decoding, batch 1, on the first 32 GSM8K and 32 HumanEval problems as single chat prompts (thinking disabled, greedy, at most 512 new tokens); means over 3 rounds. AR rows at C=64C=64 (25%), C=128C=128 (50%, the expert slots of SD) and C=148C=148 (the largest AR capacity within SD’s peak GPU memory, drafter included); rows at C≥128C\geq 128 and the compute-only AR row (no copies, forced along exact routing’s continuation) come from three further rounds. SD rows: a pool of P=64P=64 experts per layer between blocks plus 64 slots for a block’s union, k=15k=15 . ms = decode latency per generated token; copies = expert copies per generated token; τ\tau = committed tokens per verification; strict and pass@1 = scores of these generations (identical in every round). “Copies skipped” runs the same selection without any PCIe copy and gives the compute-only time; its outputs are not meaningful.
methodpredictionexecutedrouting reads
Cache-Prior–router+bonusLRU state
SpecMD (bonus)next routerrouter+bonusphysical
SMoE, MoE-COREnext routerrouter, subst.physical
Spec. Expertsnext router ∗predictednone
CommitMoEtrained MLPpredictednone
PSRnext routerrouter+bonuslogical
Table 11: How methods that change the executed experts use predictions and cache state. Prediction: of layer ll ’s experts before layer ll runs. Executed: what decides the experts that run. Routing reads: the cache state that selection depends on ( physical = filled by asynchronous copies, hence timing-dependent; logical = PSR’s cache with at most nn insertions per layer and token). subst. = substitution of missed experts.
More in this topic

Mixture-of-Experts Language Models

Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.

See all 225 papers

Explore similar work

CardsList

  1. Speculating Experts Accelerates Inference for Mixture-of-Experts

    Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts InferenceLLM Inference Acceleration

    University of Maryland · TogetherAI

  2. Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference

    Sep 4, 2026Zhenhe Wu, Yaping Jin, Qinghua Xing +6Mixture of ExpertsMemory-Efficient Optimization

    Huawei Technologies · Beihang University · Tianjin University +2

  3. Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study

    Aug 18, 2026Shriniwas Ramesh SuramMixture-of-Experts Language ModelsMixture of Experts

    University of Cumberlands, Williamsburg, USA