Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

Enda Yu; Dezun Dong; Xiangke Liao

cs.DCSep 13, 2026

Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

Authors: Enda Yu, Dezun Dong, Xiangke Liao

Organizations: National University of Defense Technology China

Abstract

Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV serves two complementary purposes---transferring to the GPU for attention computation, or computing in-place on the CPU---which demand opposing physical states. The optimal split between them varies with workload, yet existing KVCache abstractions offer only storage semantics over a monolithic object of a single physical state, and cannot express dynamic load balancing. We propose InplaceKVCache, the first KVCache abstraction whose format fixes each byte's physical residency at write time, so that the CPU--GPU load balance can be adjusted without moving data after placement. It realizes this as a four-region layout along two dimensions---device affinity and access pattern---turning load balancing into pure scheduling. Built on this abstraction, WriteScope splits CPU--GPU shares along the sequence dimension, and a portable roofline performance model determines the optimal CPU share as sequence length evolves, with online feedback tracking CPU cost drift. On three MoE models (DeepSeek-V2-Lite, Qwen3-30B-A3B, Mixtral-8 $\times$ 7B) with a 32~GB VRAM budget, WriteScope supports end-to-end inference at the 1M-token aggregate scale. In the long-context regime ( $\ge$ 8K), it achieves geometric-mean speedups of $1.5\times$ -- $2.5\times$ on A100 and $1.4\times$ -- $1.7\times$ on V100 over four reproduced baselines, while vLLM, SGLang, and KTransformers fail even with a doubled KV budget. A DeepSeek-V4-Flash case study validates composition with native sparse attention.

Figures & tables

Figure 1. Three KVCache placement strategies. WriteScope routes KV to fixed physical partitions at write time, enabling dynamic CPU-GPU load balancing without reorganization.

Model	Weights	KV	CUDA	Temp	Total
DeepSeek-V2-Lite	31.4	4.1	2.4	1.2	39.1
Qwen3-30B-A3B	61.1	12.9	2.4	1.8	78.2
Mixtral-8 $\times$ 7B	93.4	17.2	2.4	1.7	114.7

Table 1. VRAM breakdown (bs=4

\times

seq=32K, bf16, GB).

Figure 2. AF overlap contention under MoE (Qwen3-30B-A3B, bs=16, seq=32K). AF degrades attention, expert, and PCIe bandwidth simultaneously due to resource contention.

Figure 3. Strided vs. contiguous DMA on a pre-allocated buffer. Inter-row gaps fragment the source into short segments, forcing per-segment copies; a contiguous layout admits one full-bandwidth transfer.

Figure 4. InplaceKVCache four-region layout. The roofline-guided share

r

steers write-time routing of eviction batches from the GPU D-window to CPU unpinned or pinned regions. The CPU computes the unpinned share; the GPU loads the pinned share into ping-pong, with outputs merged via LSE.

Figure 5. InplaceKVCache interface. The PartitionSpec captures physical layout and roofline parameters; four thin operations enable write-time routing.

Figure 6. Pipelined sparse-selection loading. DRAM gather is hidden within the PCIe loading, leaving no extra overhead for block-level sparsity.

Figure 7. The roofline optimum

r(L)

for DeepSeek-V2-Lite and Qwen3-30B-A3B, compared with measured values.

Figure 8. Prefill throughput (2K prompts). WriteScope (Route vs. Ideal) shows write-time routing is free; mainstream frameworks OOM at scale.

Figure 9. End-to-end decode throughput. For each AF-overlap baseline, the solid and hatched bars at the same x-coordinate are its non-AF and AF variants. Mainstream frameworks are shown as points/lines (missing = OOM).

Figure 10. Ablation on DeepSeek-V2-Lite, normalized to the full system. Every other series replaces the design point it names with the baseline implementation; lower is worse.

Figure 11. Single-layer CSA latency (left) and end-to-end latency (right) in DeepSeek-V4. Lines report WriteScope’s speedup over each baseline on the right axis.

Figure 12. Deployed share

R(L)

(curves, roofline-guided) vs. the measured optimal

r

(points: per 2K of context, swept over the following 256 decode steps), on A100 and V100 (bs=16).

Figure 13. Extrapolated converged $r$ across CPU bandwidth and PCIe BW ${}_{\text{H2D}}$ ; A100 and V100 anchor the surface, and regions far from both anchors are indicative only.

Model	Ping-pong	D-window	Unpinned	Pinned
DS2	37.8 MB	255 MB	15.95 GB	16.66 GB
Qwen3	67.1 MB	805 MB	61.95 GB	41.13 GB
Mixtral	134.2 MB	1074 MB	103.08 GB	34.36 GB

Table 2. WriteScope memory footprint (seq=32K, bs=32).

Request	solo $r$	adaptive $r$	solo	fixed- $r$	adaptive
bs8/8K	0.469	0.375–0.438	499	779	651
bs8/16K	0.469	0.18–0.469	664	959	827
bs4/8K	0.438	0–0.28	386	1010	436
bs16/16K	0.547	0.18–0.547	1095	1253	1134

Table 3. Concurrent interference and adaptive mitigation on Qwen3-30B-A3B (two concurrent pairs, ms/step).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Baseline	Deviation	Favors	Rationale
SolidAttention	SSD $\rightarrow$ DRAM	baseline	removes the I/O bottleneck; PCIe streaming hits the H2D peak (25.4 GB/s)
Solid./Scout.	KV retention set to 80%	mixed	the quality-retention point reported in the original papers; keeps the comparison on transfer/compute mechanisms
HybridGen	CXL tier unavailable	neutral	502 GB host DRAM already holds the full KV; the capacity tier does not arise on this workload
FastDecode	multi-node $\rightarrow$ single-node	against	the unified single-box platform is the constant across all systems

Appendix

Table 4. Deviations from the original baselines and the direction each favors.

Baseline	Anchor	Original (reported)	Reproduction (measured)
SolidAttention	I/O bandwidth ceiling	7.5 GB/s (SSD)	25.4 GB/s (DRAM, $=$ H2D peak)
ScoutAttention	vs. full-KV attention, 8K	1.78 $\times$ (Qwen3-8B)	1.26 $\times$ e2e (same magnitude once the MoE expert share is accounted for)
HybridGen	vs. MoE-Lightning	1.87–3.02 $\times$ (OPT-13B)	1.32–3.14 $\times$ (Mixtral, native reference)
FastDecode	vs. MoE-Lightning	—	1.39–2.94 $\times$ (Mixtral, native reference)
HGCA	paradigm equivalence	—	non-AF ScoutAttention arm $=$ HGCA design (static split, sparsified CPU remainder, cross-engine merge)

Appendix

Table 5. Reproduction validation anchors. All entries are ratios or hardware ceilings, not cross-hardware absolutes.

KV-Cache Management

Key-value (KV) cache management reduces and coordinates Transformer attention state to make long-context inference faster and more memory-efficient.

See all 201 papers

Apr 3, 2026

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

Mixture-of-Experts (MoE) models have become mainstream for scaling language models to hundreds of billions of expert parameters. Despite sparse expert activation, existing inference engines keep all experts GPU-resident, crowding out the key-value cache in large-batch, long-output offline workloads. We present FluxMoE, which decouples experts from physical GPU residency and adapts their footprint to available memory through a new expert paging abstraction. FluxMoE combines PagedTensor for transparent remapping, a bandwidth-balanced hierarchy spanning losslessly compressed GPU memory and host DRAM, and a budget-aware residency planner. Unlike CPU-GPU co-inference and whole-layer offloading, FluxMoE streams weights on demand while keeping expert computation on GPUs. We implement FluxMoE atop vLLM and evaluate it on three MoE models. For GLM-4.5 on 8

\times

H20 GPUs, FluxMoE delivers up to 7.2

\times

vLLM's throughput and 79.0% lower average Time-Per-Output-Token (TPOT), without measurable model-quality loss using lossless compression. For Mixtral-8

\times

7B-Instruct on 2

\times

L40S GPUs, where weight-resident vLLM cannot fit, FluxMoE delivers 4.3

\times

KTransformers's throughput and 29.1% lower average TPOT.

Qingxiu Liu, Yongchao He, Runhan Jiang +4

CUHK · Unaffiliated · SCITIX +1

Expert Offloading GPU Acceleration Mixture of Experts Mixture-of-Experts Inference Memory-Efficient Inference LLM Inference Acceleration

Ask PDF

Jul 17, 2026

PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory requirements of the model weights and the growing KV cache. We propose PagedWeight, a novel management method for MoE LLM serving that dynamically quantizes MoE model's weights at runtime and balances expert-weight precision with the KV cache sizes. PagedWeight exposes and effectively navigates the complex tradeoff between the model's task accuracy, memory consumption, and throughput/latency. Across several memory-sensitive MoE serving scenarios, PagedWeight improves the quality-memory tradeoff over several existing quantization baselines. PagedWeight achieves FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94

\times

throughput improvement, and improves quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss.

Yuchen Yang, Yifan Zhao, Anisha Dasgupta +1

University of Illinois Urbana-Champaign

LLM Quantization Mixture-of-Experts Language Models Memory-Efficient Optimization LLM Inference Acceleration Mixture-of-Experts Quantization Weight-Only Quantization

Ask PDF

Aug 8, 2026

EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference

Load Balancing has emerged as a critical problem in expert-parallel distributed inference of Mixture-of-Experts (MoE) models. As routing distributions are typically skewed across experts, devices hosting lighter-loaded experts must idle to wait for the heaviest during expert computing, leading to inefficiency. Existing load-balancing approaches primarily rely on expert replication or migration within each layer, which introduce additional overhead and limit their flexibility and scalability. To address this problem, we propose EasyBalance, a cross-layer load balancing strategy that requires no modifications to the expert-device mapping, enabling instant adaptability and incurring essentially no additional overhead. Our key insights are that (1) experts of other layers can be viewed as naturally redundant for the current layer, and (2) cross-layer MoE workloads can be jointly executed to mitigate their individual imbalance. Based on these observations, EasyBalance greedily schedules a subset of cross-layer workloads to run at each MoE step and defers the remaining workloads for future balancing opportunities, effectively leveraging cross-layer imbalance mitigation. Extensive experiments across models, tasks, and configurations demonstrate that EasyBalance consistently accelerates distributed MoE inference, reducing GPU idling by mostly over 40%. Code is available at https://github.com/yize-wu/EasyInfra.

Yize Wu, Ke Gao, Ling Li +1

Intelligent Software Research Center, Institute of Software, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Expert Load Balancing Mixture-of-Experts Inference

Ask PDF

Abstract

KV-Cache Management

Explore similar work

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference