Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

Enda Yu; Dezun Dong; Xiangke Liao

cs.DCSep 13, 2026

Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

Authors: Enda Yu, Dezun Dong, Xiangke Liao

Organizations: National University of Defense Technology China

Abstract

Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV serves two complementary purposes---transferring to the GPU for attention computation, or computing in-place on the CPU---which demand opposing physical states. The optimal split between them varies with workload, yet existing KVCache abstractions offer only storage semantics over a monolithic object of a single physical state, and cannot express dynamic load balancing. We propose InplaceKVCache, the first KVCache abstraction whose format fixes each byte's physical residency at write time, so that the CPU--GPU load balance can be adjusted without moving data after placement. It realizes this as a four-region layout along two dimensions---device affinity and access pattern---turning load balancing into pure scheduling. Built on this abstraction, WriteScope splits CPU--GPU shares along the sequence dimension, and a portable roofline performance model determines the optimal CPU share as sequence length evolves, with online feedback tracking CPU cost drift. On three MoE models (DeepSeek-V2-Lite, Qwen3-30B-A3B, Mixtral-8 $\times$ 7B) with a 32~GB VRAM budget, WriteScope supports end-to-end inference at the 1M-token aggregate scale. In the long-context regime ( $\ge$ 8K), it achieves geometric-mean speedups of $1.5\times$ -- $2.5\times$ on A100 and $1.4\times$ -- $1.7\times$ on V100 over four reproduced baselines, while vLLM, SGLang, and KTransformers fail even with a doubled KV budget. A DeepSeek-V4-Flash case study validates composition with native sparse attention.

Figures & tables

Figure 1. Three KVCache placement strategies. WriteScope routes KV to fixed physical partitions at write time, enabling dynamic CPU-GPU load balancing without reorganization.

Model	Weights	KV	CUDA	Temp	Total
DeepSeek-V2-Lite	31.4	4.1	2.4	1.2	39.1
Qwen3-30B-A3B	61.1	12.9	2.4	1.8	78.2
Mixtral-8 $\times$ 7B	93.4	17.2	2.4	1.7	114.7

Table 1. VRAM breakdown (bs=4

\times

seq=32K, bf16, GB).

Figure 2. AF overlap contention under MoE (Qwen3-30B-A3B, bs=16, seq=32K). AF degrades attention, expert, and PCIe bandwidth simultaneously due to resource contention.

Figure 3. Strided vs. contiguous DMA on a pre-allocated buffer. Inter-row gaps fragment the source into short segments, forcing per-segment copies; a contiguous layout admits one full-bandwidth transfer.

Figure 4. InplaceKVCache four-region layout. The roofline-guided share

r

steers write-time routing of eviction batches from the GPU D-window to CPU unpinned or pinned regions. The CPU computes the unpinned share; the GPU loads the pinned share into ping-pong, with outputs merged via LSE.

Figure 5. InplaceKVCache interface. The PartitionSpec captures physical layout and roofline parameters; four thin operations enable write-time routing.

Figure 6. Pipelined sparse-selection loading. DRAM gather is hidden within the PCIe loading, leaving no extra overhead for block-level sparsity.

Figure 7. The roofline optimum

r(L)

for DeepSeek-V2-Lite and Qwen3-30B-A3B, compared with measured values.

Figure 8. Prefill throughput (2K prompts). WriteScope (Route vs. Ideal) shows write-time routing is free; mainstream frameworks OOM at scale.

Figure 9. End-to-end decode throughput. For each AF-overlap baseline, the solid and hatched bars at the same x-coordinate are its non-AF and AF variants. Mainstream frameworks are shown as points/lines (missing = OOM).

Figure 10. Ablation on DeepSeek-V2-Lite, normalized to the full system. Every other series replaces the design point it names with the baseline implementation; lower is worse.

Figure 11. Single-layer CSA latency (left) and end-to-end latency (right) in DeepSeek-V4. Lines report WriteScope’s speedup over each baseline on the right axis.

Figure 12. Deployed share

R(L)

(curves, roofline-guided) vs. the measured optimal

r

(points: per 2K of context, swept over the following 256 decode steps), on A100 and V100 (bs=16).

Figure 13. Extrapolated converged $r$ across CPU bandwidth and PCIe BW ${}_{\text{H2D}}$ ; A100 and V100 anchor the surface, and regions far from both anchors are indicative only.

Model	Ping-pong	D-window	Unpinned	Pinned
DS2	37.8 MB	255 MB	15.95 GB	16.66 GB
Qwen3	67.1 MB	805 MB	61.95 GB	41.13 GB
Mixtral	134.2 MB	1074 MB	103.08 GB	34.36 GB

Table 2. WriteScope memory footprint (seq=32K, bs=32).

Request	solo $r$	adaptive $r$	solo	fixed- $r$	adaptive
bs8/8K	0.469	0.375–0.438	499	779	651
bs8/16K	0.469	0.18–0.469	664	959	827
bs4/8K	0.438	0–0.28	386	1010	436
bs16/16K	0.547	0.18–0.547	1095	1253	1134

Table 3. Concurrent interference and adaptive mitigation on Qwen3-30B-A3B (two concurrent pairs, ms/step).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Baseline	Deviation	Favors	Rationale
SolidAttention	SSD $\rightarrow$ DRAM	baseline	removes the I/O bottleneck; PCIe streaming hits the H2D peak (25.4 GB/s)
Solid./Scout.	KV retention set to 80%	mixed	the quality-retention point reported in the original papers; keeps the comparison on transfer/compute mechanisms
HybridGen	CXL tier unavailable	neutral	502 GB host DRAM already holds the full KV; the capacity tier does not arise on this workload
FastDecode	multi-node $\rightarrow$ single-node	against	the unified single-box platform is the constant across all systems

Appendix

Table 4. Deviations from the original baselines and the direction each favors.

Baseline	Anchor	Original (reported)	Reproduction (measured)
SolidAttention	I/O bandwidth ceiling	7.5 GB/s (SSD)	25.4 GB/s (DRAM, $=$ H2D peak)
ScoutAttention	vs. full-KV attention, 8K	1.78 $\times$ (Qwen3-8B)	1.26 $\times$ e2e (same magnitude once the MoE expert share is accounted for)
HybridGen	vs. MoE-Lightning	1.87–3.02 $\times$ (OPT-13B)	1.32–3.14 $\times$ (Mixtral, native reference)
FastDecode	vs. MoE-Lightning	—	1.39–2.94 $\times$ (Mixtral, native reference)
HGCA	paradigm equivalence	—	non-AF ScoutAttention arm $=$ HGCA design (static split, sparsified CPU remainder, cross-engine merge)

Appendix

Table 5. Reproduction validation anchors. All entries are ratios or hardware ceilings, not cross-hardware absolutes.

KV-Cache Management

Key-value (KV) cache management reduces and coordinates Transformer attention state to make long-context inference faster and more memory-efficient.

See all 201 papers

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Apr 3, 2026Qingxiu Liu, Yongchao He, Runhan Jiang +4Expert Offloading GPU Acceleration
CUHK · Unaffiliated · SCITIX +1
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
Jul 17, 2026Yuchen Yang, Yifan Zhao, Anisha Dasgupta +1LLM Quantization Mixture-of-Experts Language Models
University of Illinois Urbana-Champaign
EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
Aug 8, 2026Yize Wu, Ke Gao, Ling Li +1Expert Load Balancing Mixture-of-Experts Inference
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Apr 3, 2026J/K move · Enter open · S save

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

Qingxiu Liu, Yongchao He, Runhan Jiang, Zion Wang+3

CUHK · Unaffiliated · SCITIX+1

Expert Offloading GPU Acceleration Mixture of Experts Mixture-of-Experts Inference Memory-Efficient Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models have become mainstream for scaling language models to hundreds of billions of expert parameters. Despite sparse expert activation, existing inference engines keep all experts GPU-resident, crowding out the key-value cache in large-batch, long-output offline workloads. We present FluxMoE, which decouples experts from physical GPU residency and adapts their footprint to available memory through a new expert paging abstraction. FluxMoE combines PagedTensor for transparent remapping, a bandwidth-balanced hierarchy spanning losslessly compressed GPU memory and host DRAM, and a budget-aware residency planner. Unlike CPU-GPU co-inference and whole-layer offloading, FluxMoE streams weights on demand while keeping expert computation on GPUs. We implement FluxMoE atop vLLM and evaluate it on three MoE models. For GLM-4.5 on 8

\times

H20 GPUs, FluxMoE delivers up to 7.2

\times

vLLM's throughput and 79.0% lower average Time-Per-Output-Token (TPOT), without measurable model-quality loss using lossless compression. For Mixtral-8

\times

7B-Instruct on 2

\times

L40S GPUs, where weight-resident vLLM cannot fit, FluxMoE delivers 4.3

\times

KTransformers's throughput and 29.1% lower average TPOT.

Abstract

KV-Cache Management

Explore similar work

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference