Language-model agents increasingly run on workstations with a few consumer GPUs, where large mixture-of-experts (MoE) models keep part of their experts in host memory. With existing CPU-GPU engines, these agents get fast prefill or fast decode, not both: llama.cpp's expert cache skips forward passes of more than 32 tokens to keep the experts of decode cached, so an agent waits 2.1-2.8 s after each tool output, whereas its default layer placement shortens this wait but is 1.5-1.6× slower in decode. Our key insight is that a short prefill touches too many distinct experts to upload, yet sends 84% of its expert activations (routes) to a stable set of experts that fits in GPU memory. We present Ostinato, a CPU-GPU MoE inference engine that computes these experts on the GPUs in every forward pass without changing what the model computes. Short prefills split their routes, leaving fewer than one of a token's six routed experts per layer to the CPU; long prefills upload each distinct cold expert once; and each decode step runs as one fixed-shape CUDA graph per GPU. On real coding and customer-service agent sessions with DeepSeek-V4-Flash on four RTX 4090D GPUs, Ostinato is the fastest engine in both phases and finishes every session sooner than all six baselines. Against llama.cpp and KTransformers, it cuts the median wait after a tool output by 22-55% and finishes sessions 1.4-2.5× sooner.
Figures & tables
Figure 1: Ostinato is the fastest engine both in the wait after each tool output and in the total request time. Its median wait is 1.26 s on SWE and 0.98 s on τ2 -bench, 46–55% below those of the expert caches and KTransformers and 22–24% below that of llama.cpp’s layer placement, and it finishes sessions 1.2–2.5 × sooner than every baseline. Bars: speedup over the slowest engine in each group (over FreeToken on one GPU) on DeepSeek-V4-Flash with four RTX 4090D GPUs, replaying held-out coding (SWE) and customer-service ( τ2 -bench) agent sessions.
Figure 2: Ostinato more than halves the wait after a typical tool output by computing the hot routes of its prefill on the GPU. (1) In a held-out τ2 -bench turn, the agent calls a tool, and the tool output adds 185 new tokens, which the engine prefills before the agent’s next call. (2) Each bar splits the routes of these tokens into hot and cold (84% and 16% on average in short prefills of 128 to 511 new tokens), and the streams show where each part runs. llama.cpp’s expert cache serves only forward passes of up to 32 tokens, so it computes all routes of this prefill on the CPU, including those whose experts it caches; an upload path moves every distinct cold expert over PCIe, which on our machine costs more than the CPU below 512 tokens; Ostinato computes the hot routes on the GPU and only the cold routes on the CPU. The clocks give the measured time to first token of this turn, close to the medians over all 504 τ2 -bench turns that follow a tool output (2.11 s for llama.cpp’s expert cache and 0.98 s for Ostinato ).
Figure 3: Overview of Ostinato . The resident experts run on the GPU in every forward pass; only where the cold experts run changes. Slate: hot experts and GPU; teal: cold experts and CPU; charcoal: uploads over PCIe; black numbers: the steps of Section 3.1 . (a) Offline, development sessions rank the experts (❶), a capacity test fixes the expert slots of each GPU (❷), and the highest-ranked experts fill them (❸); during decode, the buffers reserved for prefill hold borrowed experts. (b) In a decode step, the router selects k=6 experts, two of them cold (❹). Inside the CUDA graph of the GPU (dashed), the index table passes all k IDs to the GPU branch and maps the cold ones to all-zero slots, so every step keeps the same shapes. The graph writes the cold IDs and the token’s input into mapped host memory, a resident CPU thread pool outside the graph computes only the r=2 cold experts (❺), and a wait kernel joins the two branches before the next layer (❻). The timeline is drawn to measured scale for r=2 . (c) A prefill forward pass of fewer than Np=512 tokens splits its routes between the two branches, which run in turn; a larger one uploads each of its distinct cold experts once (❼).
Model
Date
Params
Experts ( k )
Format
DeepSeek-V4-Flash
2026-07
284B/13B a
43 × 256 (6)
MXFP4
Qwen3-235B-A22B
2025-07
235B/22B
94 × 128 (8)
Q4_K_M
Table 1: Evaluated models. Date: release month; Params: total / active; Experts ( k ): MoE layers × routed experts in each layer, and in parentheses the number that each token selects. DeepSeek-V4-Flash also has one shared expert per layer. a Without its speculative-decoding module.
Set
Sessions
Turns
Outputs
New
Software engineering (SWE)
development
6
400
105,879
258
screening a
2
60
9,842
335
held-out
13
893
215,341
249
τ2 -bench
development
24
335
32,931
135
Table 2: Replayed workloads. Outputs: generated tokens; New: median number of tokens prefilled per turn after reuse of the previous turn’s KV cache. a The first 30 turns of two development sessions, used for ablations.
SWE (13 sessions)
τ2 -bench (60 tasks)
Engine
Time (s)
Saved
Won
TTFT (s)
TPOT (ms)
Time (s)
Saved
Won
TTFT (s)
TPOT (ms)
KTransformers
16,950.1
58.0% 57.6–58.4
13/13
2.36
62.8
7,739.6
60.4% 59.9–61.0
60/60
1.88
61.6
llama.cpp (layers)
11,880.4
40.1% 39.3–40.7
13/13
1.66
47.1
5,073.9
39.6% 38.9–40.3
60/60
1.25
46.7
llama.cpp expert cache
9,638.7
26.1% 25.4–26.9
13/13
2.78
31.5
4,357.2
29.7% 28.9–30.6
60/60
2.11
30.0
ik_llama.cpp
11,207.8
36.5% 36.0–37.0
13/13
2.61
39.6
4,795.4
36.1% 35.5–36.9
60/60
2.08
36.5
buun (llama.cpp fork)
8,523.9
16.5% 15.5–17.4
13/13
2.37
27.3
3,846.2
20.4% 19.6–21.2
60/60
1.80
26.0
Table 3: Ostinato has the shortest total request time, time to first token and time per output token on both workloads. Held-out results on DeepSeek-V4-Flash. Time: total request time of the 13 SWE sessions or the 60 τ2 -bench tasks; Saved: share of that time that Ostinato saves, with its 95% confidence interval below; Won: sessions or tasks that Ostinato finishes sooner; TTFT: median time to first token after a tool output; TPOT: median time per output token. Best values in bold.
Figure 4: Ostinato finishes every held-out SWE session and τ2 -bench task sooner than every baseline on DeepSeek-V4-Flash, saving at least 12.4% of the request time in each. Points: request time that Ostinato saves in each session or task against each baseline (FreeToken: one GPU, SWE only); labels: median saving.
Figure 5: Ostinato has the shortest median wait after a tool output on both workloads, 55% and 54% below that of llama.cpp’s expert cache. Time to first token (TTFT) of every turn that follows a tool output on DeepSeek-V4-Flash: all 880 such turns of the held-out SWE sessions and 504 of the held-out τ2 -bench tasks. Curves further left mean shorter waits; circles: medians of llama.cpp’s expert cache and Ostinato ; arrows: the change of the median. The axis ends at 30 s, beyond which fewer than 1% of any engine’s turns lie.
Figure 6: Ostinato spends less time than every llama.cpp-based baseline in both prefill and decode, has the shortest median wait after tool outputs and user messages, and decodes fastest at every context length. (a, b) Prefill (dark) and decode (light) time of all held-out sessions on DeepSeek-V4-Flash as reported by each engine; dashed lines: prefill time and total of Ostinato . (c) Median time to first token after a tool output and after a user message (first turns in Table 7 ). (d) Median time per output token of the held-out SWE turns in quartiles of context length, plotted at the median context of each quartile; per turn, the time from the first token to the end of the request divided by the output tokens minus one.
Figure 7: Below 512 new tokens, where Ostinato takes the split path, it waits least in every group, its wait grows about half as fast with input length as that of buun, and its CPU computes 6.3–6.5 × fewer experts per token. (a) Median time to first token of the 1,384 held-out turns that follow a tool output on DeepSeek-V4-Flash (880 SWE and 504 τ2 -bench turns), grouped by new tokens (fewer than 128, 128–255, 256–511, 512–1,023 and at least 1,024) and plotted at the median length of each group. (b) Prefilling 128 to 1,024 new tokens after eight development prefixes, against buun, which switches from computing experts on the CPU to uploading them at the same 512-token threshold; labels: least-squares slope on the split path. (c) Experts that the CPU computes per token and layer in prefills of 128, 343 and 511 new tokens after the same prefixes: all six for llama.cpp’s expert cache and buun, whose caches skip forward passes of more than 32 tokens, and 0.92–0.95 for Ostinato .
Figure 8: The prefill paths nearly halve the prefill time, and the decode design cuts a decode step from 29.4 to 25.6 ms on SWE, below buun’s 28.7 ms. Ablation on DeepSeek-V4-Flash; below the references, each row adds one component to the row above. (a) Screening set (60 turns) and (b) held-out SWE sessions (893 turns): prefill and decode time of each configuration, labeled with its total time. Panels (a) and (b) decode with the two branches and llama.cpp’s per-segment CUDA graphs, without the decode graph and borrowed memory of (c), with 5,702 resident experts in (a) and 4,864 growing to 5,702 in the last row of (b); with the full decode path, the sessions of (b) take the 7,119.0 s of Table 3 . (c) Time per decode step on two development prefixes per workload, which isolates the decode components, with buun as the reference; the prefill of each workload changes by less than 0.2 s across the rows. Appendix D lists all numbers.
Turn type
Turns
TTFT (s)
Change
after a tool output
504
2.75 → 1.67
−39.4%
after a user message
352
1.28 → 0.54
−57.8%
total request time
916
−16.2%
Table 4: On Qwen3-235B-A22B, the general residency and prefill paths cut the median time to first token by 39–58% and the total request time by 16.2% against buun. Median time to first token by turn type on the 60 held-out τ2 -bench tasks, buun → Ostinato , and the total request time.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
E , k
routed experts of an MoE layer; experts that a token selects
Lj , κj
MoE layers on GPU j ; expert slots of GPU j
Hℓ , H
resident (hot) experts of layer ℓ ; the residency set of all layers
rℓ
cold routes of layer ℓ in a decode step
Gℓ
time of the GPU branch of layer ℓ in a decode step
C(r)
time of the CPU branch with r cold routes: γ0 if r=0 , else α+βr
Appendix
Table 5: Notation of the cost model.
Engine
GPUs
Experts on the GPUs
Other settings
Ostinato
4
5,702 most frequently routed experts in a static pool, plus 544 slots of borrowed memory during decode
prefill threshold Np of 512 tokens
buun
4
dynamic cache of automatic size that admits experts by decaying access counts, warmed with a development routing profile
upload threshold of 512 tokens chosen on a development session; CPU–GPU overlap on
llama.cpp expert cache
4
74,048 MiB cache shared by the layers of each device
largest budget that passes the capacity test; dense weights placed by llama.cpp’s automatic fit
ik_llama.cpp
4
69,632 MiB cache
budget chosen on development sessions
KTransformers
4
58 experts per layer in the default uniform placement, with dynamic expert updates
tensor parallelism over 4 GPUs; 48 CPU inference threads in two pools; prefill in chunks of 2,048 tokens; fastest of seven configurations on a development session
llama.cpp (layers)
4
whole MoE layers placed by llama.cpp’s automatic fit, with 2,048 MiB left free on each GPU
no expert cache; forward passes of 512 or more tokens offload the CPU layers to the GPUs (threshold chosen on a development session)
Appendix
Table 6: Engine configurations on DeepSeek-V4-Flash. All engines use a 98,304-token context, one slot and YaRN RoPE scaling with factor 16; the four-GPU llama.cpp-based engines run 48 threads on 48 cores with a batch of 4,096 and a micro-batch of 2,048 tokens.
SWE
τ2 -bench
Engine
first (13)
tool (880)
first (60)
tool (504)
user (352)
KTransformers
32.60
2.36
7.84
1.88
1.63
llama.cpp (layers)
4.57
1.66
3.82
1.25
0.47
llama.cpp expert cache
6.72
2.78
5.64
2.11
0.52
ik_llama.cpp
10.62
2.61
6.10
2.08
0.65
buun (llama.cpp fork)
5.82
2.37
5.33
1.80
0.73
Appendix
Table 7: Ostinato has the shortest median time to first token after tool outputs and user messages; on first turns, llama.cpp’s layer placement is 0.08–0.12 s faster. Median time to first token (s) by turn type on DeepSeek-V4-Flash, held-out sessions: the first turn of each SWE session or τ2 -bench task, the turns that follow a tool output and the τ2 -bench turns that follow a user message (number of turns in parentheses). Best values in bold.
(a) Prefill and residency
Configuration
Total (s)
Prefill (s)
Decode (s)
Screening set (60 turns)
layer placement
629.1
165.6
463.7
llama.cpp expert cache
571.6
261.9
308.7
resident experts (5,702), native prefill
581.0
284.8
295.2
+ size-dependent prefill paths
447.7
150.8
295.8
Appendix
Table 8: Ablation numbers behind Figure 8 ; each row adds one component to the row above. (a) Prefill and residency: total, prefill and decode time of the 60-turn screening set and of the 893 held-out SWE turns; these rows use the decode path of an earlier version, and the total also counts the time outside prefill and decode. (b) Decode: time per decode step on two development prefixes per workload; the prefill of each workload changes by less than 0.2 s across the rows.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.
Mixture-of-Experts (MoE) models reduce inference computation through sparse expert activation, but their large expert memory footprint often exceeds GPU capacity, making expert offloading necessary for memory-constrained inference. However, existing systems typically handle GPU-missed experts by either transferring them to the GPU, incurring substantial PCIe transfer latency, or executing them on the CPU, making the slower CPU computation the bottleneck. To address these limitations, we present \method, a CPU-GPU collaborative MoE inference system that adaptively assigns these missed experts to the CPU or GPU for efficient execution. By exploiting differences in expert token-loads, \method selectively transfers high-load experts to the GPU while executing low-load experts on the CPU. To minimize inference latency, \method adaptively adjusts expert assignments to balance the CPU computation path against the GPU transfer-and-computation path, allowing both execution paths to complete as early as possible. Extensive experiments on three representative MoE LLMs show that \method improves prefill speed by up to 51.9% and decode throughput by up to 39.5% over the strongest baseline across the evaluated batch sizes, with consistent gains under different GPU-memory budgets.
The Mixture of Experts (MoE) architecture has become a fundamental building block in state-of-the-art large language models (LLMs), improving domain-specific expertise in LLMs and scaling model capacity without proportionally increasing their computational overhead. However, MoE inference often suffers from suboptimal GPU utilization, load imbalance, and elevated latency arising from multiple tokens waiting on the same experts for their computation which arises from sparsity of expert activation. To address these challenges, we propose a dynamic expert replication strategy that predicts which experts are likely to be overloaded and replicates them for upcoming batches of tokens. The replicated experts process batch tokens concurrently across layers, which leads to improved parallelism, shorter GPU idle time, and significantly faster inference. Experimental evaluations conducted on large-scale MoE models, including Switch-base-128 and Switch-base-256, demonstrate that our method achieves near-complete GPU utilization (approx 100%), leading to upto 3x improvement in inference speed while preserving approximately 90-95% of the performance of baseline architectures
Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar +1