Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC

Mohammad Siavashi; Mariano Scazzariello; Gerald Q. Maguire; Dejan Kostić; Marco Chiesa

cs.DCApr 8, 2026

Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC

Authors: Mohammad Siavashi, Mariano Scazzariello, Gerald Q. Maguire, Dejan Kostić, Marco Chiesa

Organizations: KTH Royal Institute of Technology, Stockholm, Sweden · RISE, Stockholm, Sweden

Abstract

Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized. We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement. Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47 $\times$ and P99 TPOT by up to 3.40 $\times$ , improving decode throughput by up to 2.1 $\times$ , and reducing energy per token by up to 48.6 $\%$ . Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude.

Figures & tables

Figure 1. Achieved throughput of 4 LLM serving systems on Qwen-3 30B-A3B (MoE) at 4 req/s offered load, under isolated and colocated execution. Blink is unaffected by colocation while baselines degrade. Annotations show the colocated-to-isolated throughput ratio.

	Baseline	Interference
		12 $\times$	24 $\times$
Throughput (tok/s)	$7\,475$	$4\,554$	$1\,961$
Mean TTFT (ms)	$73.7$	$4\,865$	$16\,552$
P99 TTFT (ms)	$150$	$6\,366$	$20\,959$
Mean TPOT (ms)	$13.0$	$13.6$	$14.8$
P99 TPOT (ms)	$14.4$	$18.0$	$32.1$

Table 1. Impact of colocation on vLLM serving latency and microarchitectural counters. H100, Llama-3 8B,

7\text{\,}\mathrm{r}\mathrm{e}\mathrm{q}\mathrm{/}\mathrm{s}

, w/ CUDA Graphs.

	4 KB pages	2 MB pages	1 GB (interferer)
Throughput (tok/s)	$4\,813$	$5\,053$	$4\,5405\,298$
P50 TTFT (ms)	$12\,276$	$12\,612$	$12\,57815\,205$
P99 TTFT (ms)	$29\,208$	$26\,780$	$29\,19132\,270$
P50 TPOT (ms)	$19.8$	$19.9$	$17.919.6$
P99 TPOT (ms)	$25.0$	$25.4$	$27.230.5$
P99 ITL (ms)	$70.1$	$72.2$	$73.284.8$

Table 2. Ablation: page size and gigantic pages do not restore isolation (H100, CUDA Graphs enabled, 128 requests at

7\text{\,}\mathrm{r}\mathrm{e}\mathrm{q}\mathrm{/}\mathrm{s}

). All runs include interference. Isolation baseline:

7\,697\text{\,}\mathrm{t}\mathrm{o}\mathrm{k}\mathrm{/}\mathrm{s}

throughput,

13.5\text{\,}\mathrm{ms}

mean TPOT ,

5.9\text{\,}\mathrm{\%}

LLC miss rate.

	Isolation	Interference	$\Delta$ %
Total completed requests	$502$	$415$	-17.3 %
Mean throughput (tok/s)	$4\,130.50$	$3\,457.85$	-16.3 %
Mean throughput (req/s)	$8.37$	$6.92$	-17.3 %
P50 TTFT (ms)	$10\,052.08$	$12\,539.18$	+24.7 %
P99 TTFT (ms)	$22\,409.94$	$23\,968.67$	+7.0 %
P99.9 TTFT (ms)	$23\,925.64$	$25\,732.75$	+7.6 %

Table 3. Core pinning (6 dedicated cores) under pbzip2 interference. ShareGPT workload, Poisson arrivals at

12\text{\,}\mathrm{r}\mathrm{e}\mathrm{q}\mathrm{/}\mathrm{s}

60\text{\,}\mathrm{s}

window.

Cache ways	$1$	$3$	$5$	$7$	$12$
LLC miss rate (%)	$57.6$	$26.6$	$11.1$	$7.0$	$6.8$
IPC	$1.16$	$1.31$	$1.48$	$1.52$	$1.55$
LLC stall cycles	$3\,169$ M	$1\,750$ M	$741$ M	$428$ M	$442$ M
dTLB load misses	$7.4$ M	$7.1$ M	$7.1$ M	$7.0$ M	$7.0$ M
walk_active	$895$ M	$596$ M	$462$ M	$420$ M	$400$ M
P99 TTFT (ms)	$29\,675$	$26\,627$	$27\,146$	$26\,352$	$26\,157$

Table 4. Effect of CAT cache-way allocation under interference (H100, CUDA Graphs,

128

requests at

7\text{\,}\mathrm{r}\mathrm{e}\mathrm{q}\mathrm{/}\mathrm{s}

, dedicated CPU cores). LLC miss rate drops

8.5\times

, yet P99 ITL is virtually unchanged.

Figure 2. Architecture of Blink . After one-time initialization, the host CPU is idle; the steady-state inference path involves only the GPU and the DPU. Filled circles trace the lifecycle of a single request.

Figure 3. Normalized makespan: GPU-resident vs. CPU-resident scheduling on Qwen3-32B (N

\times

\rightarrow

O denotes N requests, I input tokens, O output tokens). CPU vs. GPU-resident scheduler makespan Bar chart comparing normalized makespan. The CPU-resident scheduler is 1.16–1.70x slower across four workload configurations.

Figure 4. Tokenization latency on BlueField-3 ARM A78 cores ( Blink ) vs. Intel Xeon (HuggingFace, llama.cpp). Labels show speedup relative to HuggingFace. Tokenization latency comparison Bar chart comparing tokenization latency for HuggingFace, llama.cpp, and \systemnameacross input lengths of 10–2048 tokens. \systemnameis 8–19.7x faster than HuggingFace.

Component	Specification
GPU	NVIDIA H100 (96 GB HBM3)
CPU	2 $\times$ Intel Xeon Gold 6336Y (96 cores @ 2.40 GHz),
	DVFS disabled, governor: performance
DRAM	256 GB DDR5
Network	ConnectX-6 (200 Gbps)
DPU	BlueField-3 (16 ARM Cortex-A78, 32 GB)

Table 5. Hardware configuration.

Model	System	Geo. P99 TTFT	Geo. P99 TPOT	Tput at sat.
Llama-3 8B $\lambda\leq 12$	Blink	653.8	15.1	11.87
	TRT-LLM	880.0	17.7	10.80
	vLLM	1309.6	24.2	9.12
	SGLang	1747.1	30.7	7.88
Phi-4 15B $\lambda\leq 7$	Blink	1109.4	25.0	6.72
	TRT-LLM	1453.8	29.8	6.42

Table 6. Pre-saturation summary over the Blink -defined operating range. Repeated runs are averaged per load to match the figures. TTFT and TPOT are in milliseconds; Tput@sat is achieved throughput at Blink ’s saturation point. Best values per model are boldfaced.

Panel of Figure 5 — Figure 5. P99.9 tail latency for Qwen-3 32B (isolated). Although P99 latencies are near-parity, P99.9 reveals a consistent Blink advantage across all baselines that grows with load. P99.9 TTFT and TPOT for Qwen-3 32B P99.9 TTFT and TPOT latency comparison between \systemname, TensorRT-LLM, vLLM, and SGLang on Qwen-3 32B under isolated execution.

Panel of Figure 6 — Figure 6. P99 tail latency across four models. Top row (a–d): TTFT; bottom row (e–h): TPOT. Solid lines show isolated execution; dashed lines show CPU interference. Within Blink ’s operating range, Blink maintains a flatter latency envelope and near-overlapping isolated and interference curves, whereas baseline systems queue earlier and suffer severe tail inflation under co-location. P99 TTFT and TPOT under isolated and interference conditions P99 TTFT and TPOT latency across model architectures under both isolated and interference conditions.

Model	System	Geo. P99 TTFT	Geo. P99 TPOT	Tput at sat.
Llama-3 8B $\lambda\leq 12$	Blink	652.8 [1.00]	15.2 [1.00]	11.83 [1.00]
	TRT-LLM	16574 [18.84]	196.6 [11.10]	4.13 [0.38]
	vLLM	14563 [11.12]	178.2 [7.35]	4.00 [0.44]
	SGLang	14728 [8.43]	177.3 [5.77]	3.78 [0.48]
Phi-4 15B $\lambda\leq 7$	Blink	1023.2 [0.92]	24.4 [0.98]	6.77 [1.01]
	TRT-LLM	15496 [10.66]	183.7 [6.17]	2.62 [0.41]

Table 7. Same pre-saturation summary as Table 6 , but under CPU interference. Bracketed terms show interference / isolation; throughput brackets report retention at Blink ’s saturation point. Raw latencies are in milliseconds and throughput in requests/s. Best raw values per model are boldfaced.

Panel of Figure 7 — Figure 7. Throughput across four models. Solid lines show isolated execution; dashed lines show CPU interference. Blink reaches the latest or tied-latest saturation point in isolation and preserves its plateau under interference, while baselines either saturate earlier or collapse to much lower plateaus. Throughput under isolation and interference for all four models Throughput under isolation and interference for all four models.

Figure 8. Energy per token under (a) isolation and (b) colocated interference. Energy per token for four models under isolation and interference Bar chart comparing energy per token (mJ/tok) for \systemname, vLLM, TensorRT-LLM, and SGLang across LLaMA 3 8B, Phi-4, Qwen3 30B-A3B, and Qwen3 32B under isolation and co-located interference.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	System	P50 TTFT	Mean TTFT	P50 TPOT	Mean TPOT
Llama-3 8B	Blink	41.8	116.9	7.5	8.2
	TRT-LLM	72.3	170.1	8.6	9.6
	vLLM	111.1	276.1	11.0	12.4
	SGLang	240.3	507.1	12.9	14.5
Phi-4 15B	Blink	105.8	258.8	13.4	14.1
	TRT-LLM	153.1	355.7	15.2	16.3

Appendix

Table B.1 . Geometric-mean latency (ms) over Blink ’s operating range under isolated execution. Best values per model are boldfaced. Operating ranges:

\lambda\leq 12

(Llama-3 8B),

\leq 7

(Phi-4 15B),

\leq 2

(Qwen-3 32B),

\leq 4

(Qwen-3 30B-A3B).

Model	System	Decode tok/s	Prefill tok/s
Llama-3 8B	Blink	3880	595
	TRT-LLM	3535	582
	vLLM	2930	564
	SGLang	2638	553
Phi-4 15B	Blink	2177	465
	TRT-LLM	2044	459

Appendix

Table B.2 . Token-level throughput (tok/s) at Blink ’s saturation point under isolated execution. Best values per model are boldfaced.

Panel of Figure 18 — Figure C.1 . Maximum serviceable load (95% throughput retention threshold) per system and model. Higher is better. Blink achieves the highest serviceable load on every model; on Qwen-3 30B-A3B the gap is largest under isolation, and under interference Blink retains its full isolated capacity while baselines lose 60–65% on Llama-3 8B. Serviceable load bar charts under isolated and interference Two bar charts showing maximum serviceable load for all systems across four models: (a) isolated and (b) interference conditions.

Panel of Figure 19 — Figure D.1 . P99.9 tail latency across four models. Top row (a–d): TTFT; bottom row (e–h): TPOT. Solid lines show isolated execution; dashed lines show CPU interference. Blink sustains lower P99.9 latency across the full load sweep on all four models, with the largest margin on the MoE model (Qwen-3 30B-A3B). Baseline dashed curves separate sharply from their isolated counterparts, while Blink ’s remain nearly overlapping. P99.9 latency for all models P99.9 TTFT and TPOT across four models under isolated and interference conditions.

Panel of Figure 20 — Figure D.2 . P95 latency across four models. Top row (a–d): TTFT; bottom row (e–h): TPOT. Solid lines show isolated execution; dashed lines show CPU interference. The relative system ordering is consistent with the P99 results in the main paper, confirming that Blink ’s advantage is structural rather than an artifact of rare outliers. P95 latency for all models P95 TTFT and TPOT across four models under isolated and interference conditions.

Panel of Figure 21 — Figure D.3 . Median (P50) latency across four models. Top row (a–d): TTFT; middle row (e–h): TPOT; bottom row (i–l): ITL. Solid lines show isolated execution; dashed lines show CPU interference. Blink achieves the lowest or near-lowest median across all metrics, with differences compressing on the GPU-bound Qwen-3 32B. Under interference, baseline medians inflate substantially while Blink ’s remain stable. P50 latency for all models P50 TTFT, TPOT, and ITL across four models under isolated and interference conditions.

Panel of Figure 22 — Figure D.4 . Mean latency across four models. Top row (a–d): TTFT; middle row (e–h): TPOT; bottom row (i–l): ITL. Solid lines show isolated execution; dashed lines show CPU interference. The mean tracks between P50 and P95 due to heavy-tailed queuing distributions; system ordering remains consistent with tail percentile results. Baseline means inflate by 3–18 $\times$ under interference while Blink ’s stays within 1.0–1.15 $\times$ . Mean latency for all models Mean TTFT, TPOT, and ITL across four models under isolated and interference conditions.

Panel of Figure 23 — Figure E.1 . Token-level throughput across four models. Top row (a–d): prefill throughput (prompt tokens/s); bottom row (e–h): decode throughput (output tokens/s). Solid lines show isolated execution; dashed lines show CPU interference. Blink ’s GPU-resident scheduling eliminates per-iteration CPU round-trips, yielding higher sustained decode throughput. Under interference, baseline decode throughput collapses while Blink ’s plateau is preserved. Prefill and decode throughput for all models Prefill and decode token throughput across four models under isolated and interference conditions.

Explore similar work

ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs
Mar 8, 2026Yuzhuang Xu, Xu Han, Yuxuan Li +1Efficient Neural Network Inference LLM Inference Acceleration
Harbin Institute of Technology, Harbin, China · Tsinghua University, Beijing, China
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
Jul 30, 2026Jiaxuan Chen, Jianshu She, Ye Yuan +5LLM Serving LLM Fine-Tuning
McGill University · Canada · MBZUAI +3
Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model Inference LLM Inference
MBZUAI · UESTC · City University of Hong Kong +1

Mar 8, 2026J/K move · Enter open · S save

ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

Yuzhuang Xu, Xu Han, Yuxuan Li, Wanxiang Che

Harbin Institute of Technology, Harbin, China · Tsinghua University, Beijing, China

Efficient Neural Network Inference LLM Inference Acceleration LLM Inference Scheduling

Open paper AskPDF arXiv

Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.

Abstract

Explore similar work

ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters