DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving

YuXuan Lin; KeQiang Wu; Kingsum Chow

cs.DCOct 24, 2026

DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving

Authors: YuXuan Lin, KeQiang Wu, Kingsum Chow

Organizations: Ningbo Global Innovation Center, Zhejiang University, Ningbo, China

Abstract

Large language model (LLM) serving relies on batching to improve inference throughput, but the key-value (KV) cache grows with sequence length and limits GPU-resident concurrency. CPU offloading can expand KV capacity using host memory, yet existing systems remain largely GPU-gated: CPU-resident requests make progress only when the GPU scheduler exposes suitable overlap opportunities. As a result, host memory and CPU compute cannot act as an independent serving resource, leading to bursty CPU-side progress and inflated tail inter-token latency (ITL) under load. We present DualPace, an LLM serving runtime that turns the host into an independently scheduled decode engine. DualPace separates scheduling authority across two engines: the GPU engine executes prefills and GPU-resident decodes, while an AMX-enabled CPU engine independently batches and advances CPU-resident decodes. The engines are decoupled in scheduling but cooperative in execution: the CPU can execute a complete decode iteration locally or selectively invoke GPU Dense Assist for dense stages, while offline profiles guide hardware-specific batching and routing. Our evaluations show that, in decode-oriented scenarios, DualPace improves serving throughput by 6.9--43.2% over NEO, a state-of-the-art open-source CPU-offloading system, and by 11.5--90.6% over vLLM, a widely used GPU serving system, while maintaining the same level of latency. DualPace also preserves NEO's P99 time-to-first-token (TTFT) advantage over vLLM while sustaining 7.3--50.5% higher request rates than NEO at the same level of P99 ITL.

Figures & tables

Figure 1. Llama-3.1-8B runs on an RTX 4090 with 12 physical Intel Xeon Silver 4510 cores, serving 2,000 Azure-Conversation requests ( Stojkovic et al., 2025 ; Microsoft Azure, 2024 ) . Top left: cumulative generated tokens; DualPace sustains the steepest slope and completes the trace first. Top right: cumulative CPU-resident decoded tokens; DualPace advances continuously, whereas NEO exhibits repeated plateaus. The inset magnifies a representative progress stall. Bottom: P99 client-observed inter-token latency (ITL) across offered loads, quantifying the tail-latency consequence of intermittent CPU-resident progress. Cumulative total-token and CPU-resident token progress, followed by P99 inter-token latency across request rates.

Figure 2. Conventional decoder-only LLM inference and the analytical execution stages used throughout this paper.

Figure 3. Latency of representative Llama-3.1-8B operations on a 12-core Intel Xeon Silver 4510 CPU (Sapphire Rapids) and an RTX 4090 GPU ( Intel Corporation, 2023 ; NVIDIA, 2022 ) . Top: PagedAttention latency. Bottom: packed-QKV and FFN gate/up projection latency.

Figure 4. NEO’s asymmetric GPU–CPU pipeline, redrawn from ( Jiang et al., 2025b ) . Subscripts 0 and 1 identify sub-batches. Timeline of NEO's asymmetric GPU–CPU pipelining mechanism.

Figure 6. DualPace’s decoupled GPU–CPU pipeline with stage-boundary cooperation.

Figure 7. Profile-guided configuration decision visualization chart for Llama-3.1-8B on a 12-core Intel Xeon Silver 4510 CPU and an NVIDIA RTX 4090 GPU platform.

Figure 8. Prompt and output length distributions of the Code and Conversation workloads in the Azure LLM Inference Trace ( Stojkovic et al., 2025 ; Microsoft Azure, 2024 ) : (a) Code and (b) Conversation.

Figure 9. Normalized latency under increasing offered load on the Azure-Conversation and Azure-Code workloads.

Figure 10. Tail latency under increasing offered load on the Azure-Conversation workload.

Figure 11. Sensitivity to CPU tensor parallelism and KV capacity. Sensitivity curves for CPU tensor parallelism and host KV-cache capacity across normalized latency, P99 TTFT, and P99 ITL.

Figure 12. GPU Dense Assist component-time breakdown. Left: normalized host-orchestration and CUDA component times. Right: CUDA-time decomposition. Normalized host and CUDA component times for GPU Dense Assist, followed by a CUDA-time breakdown into transfer, post-attention, and sampling stages.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 13. P99 client-observed TTFT and ITL under increasing offered load on the Azure-Code workload. Six line charts comparing DualPace, NEO, and vLLM. The top row shows P99 time to first token on Azure-Code, and the bottom row shows P99 inter-token latency for three model–server configurations.

Figure 14. P99 client-observed TPOT under increasing offered load on the Azure-Conversation and Azure-Code workloads. Six line charts comparing P99 time per output token for DualPace, NEO, and vLLM. The top row uses Azure-Conversation and the bottom row uses Azure-Code across three model–server configurations.

Figure 15. P99 client-observed TPOT sensitivity on Azure-Conversation for Llama-3.1-8B on one RTX 4090. Left: CPU tensor parallelism, with 12 physical cores per rank and the same aggregate host KV-cache capacity. Right: host KV-cache capacity. The horizontal dashed line marks the

1\,\mathrm{s}

P99 TPOT target; vertical dashed lines mark interpolated sustainable request rates when the target is crossed within the measured sweep. Two line charts showing P99 time per output token under increasing offered load. The left chart compares CPU tensor parallelism degrees one and two. The right chart compares host KV-cache capacities of 30, 60, 120, and 240 GiB.

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 216 papers

Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
Apr 8, 2026Mohammad Siavashi, Mariano Scazzariello, Gerald Q. Maguire +2LLM Serving LLM Inference
KTH Royal Institute of Technology, Stockholm, Sweden · RISE, Stockholm, Sweden
PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs
Jun 25, 2026Muhammad AhmedLLM Serving GPU Kernel Optimization
PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
Sep 28, 2026Qiuyang Zhang, Kai Zhou, Kai Lu +6KV-Cache Offloading Long-Context Language Model Inference
Huazhong University of Science and Technology Wuhan, China · UCloud Technology Co., Ltd. Shanghai, China

Apr 8, 2026J/K move · Enter open · S save

Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC

Mohammad Siavashi, Mariano Scazzariello, Gerald Q. Maguire, Dejan Kostić+1

KTH Royal Institute of Technology, Stockholm, Sweden · RISE, Stockholm, Sweden

LLM Serving LLM Inference LLM Inference Efficiency LLM Inference Scheduling

Open paper AskPDF arXiv

Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized. We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement. Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47

\times

and P99 TPOT by up to 3.40

\times

, improving decode throughput by up to 2.1

\times

, and reducing energy per token by up to 48.6

\%

. Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude.

Abstract

LLM Serving

Explore similar work

Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC

PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs

PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding