TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition

Prabhu Vellaisamy; Shreesh Tripathi; Vignesh Natarajan; Surya Santhan Thenarasu; Shawn Blanton; John P. Shen

cs.DCMar 12, 2026

TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition

Authors: Prabhu Vellaisamy, Shreesh Tripathi, Vignesh Natarajan, Surya Santhan Thenarasu, Shawn Blanton, John P. Shen

Organizations: NeuroAI Computer Architecture Lab (NCAL), Electrical and Computer Engineering Department, Carnegie Mellon University

Abstract

Large Language Model (LLM) inference is widely used in interactive assistants and agentic systems. In latency-sensitive deployments, inference time can become dominated by host-side overheads. Existing approaches typically expose this cost only as an aggregate residual or a launch/queue metric, which is often insufficient to identify which execution layer should be optimized. This work presents TaxBreak, a trace-driven methodology for decomposing host-visible orchestration overhead into three components: framework translation time, CUDA library translation time, and kernel launch-path time. We validate TaxBreak on NVIDIA H100 and H200 systems and use it to derive our proposed Host-Device Balance Index (HDBI), a boundedness summary index that relates device-active execution to host-visible orchestration. Across representative dense and mixture-of-experts workloads in both prefill and decode, we show that aggregate latency, GPU inactivity, or boundedness ratios alone can obscure the dominant optimization target. TaxBreak instead distinguishes cases where optimization should reduce software-stack overhead from cases where the primary win comes from reducing device-side work. We further show that MoE models dispatch 8-11x more kernels per output token than dense models, and that for such host-bound workloads, CPU single-thread performance is a first-order parameter: a faster host CPU reduces orchestration overhead by 10-29% and improves end-to-end latency by up to 14%, even when paired with a slower-clocked GPU. These results position TaxBreak as a diagnostic tool for assessing whether optimization effort should target the software stack or the device-side workload execution.

Figures & tables

Fig. 1 : TaxBreak Methodology. TaxBreak decomposes overall host-side orchestration overhead into three components: (i) framework execution, (ii) CUDA library front-end execution, and (iii) kernel launch invocation. We also introduce a new Host-Device Balance Index (HDBI) to characterize the relative boundedness between the host (CPU) and the device (GPU). Prior work A refers to aggregate framework tax [ 14 ] ; B refers to kernel launch/queue tax (TKLQT) [ 30 ] .

Fig. 2 : Previous characterizations of GPT-2 inference across batch sizes. Left: end-to-end latency (ms) shows a transition from framework-bound to compute-bound at small batch sizes [ 14 ] . Right: TKLQT in

\mu

s highlights similar CPU-bound to GPU-bound transition as kernel queuing increases with batch size and utilization [ 30 ] .

Work	Tax Granularity	CPU-GPU	Cross-Layer Overhead	Workloads	Prefill + Decode	Hopper-Era HW	Evaluation Platform(s)
AI Tax [ 25 ]	Pipeline-level	✗	✗	Google FaceNet, R-CNN	✗	✗	Edge data center deployment
Framework Tax [ 14 ]	Coarse residual	✗	✗	BERT-Base; ResNet-50; DistilBERT; GPT-2; WavLM.	✗	✗	NVIDIA Pascal, Turing, Ampere.
TKLQT / Kernel-Tax View [ 30 ]	Launch-path only	✓	✗	Bert-Base-Uncased; XLM-Roberta-Base; GPT-2; Llama-3.2-1B.	✗	✓	PCIe A100/H100 and tightly-coupled GH200 (prefill-centric).
GPU Inference Characterization [ 31 ]	Device-centric	✗	✗	Llama-3-8B; Qwen2.5-7B/-32B; Qwen3-30B-A3B.	✓	✗	$4\times$ NVIDIA A100 80GB (SXM) + Jetson AGX Orin.
This Work (TaxBreak)	Host-stack attribution: $\Delta{FT}$ , $\Delta{CT}$ , $\Delta{KT}$	✓	✓	Llama-3.2-1B,-3B, OLMoE-1B/7B; Qwen1.5-MoE-A2.7B	✓	✓	H100 (DGX H100); H200 NVL (single-GPU).

TABLE I : Comparisons with Previous Works

(a) Library-mediated	(b) Framework-native
torch_op	torch_op
(nn.Linear.call)	(layer_norm)
$\downarrow$	$\downarrow$
aten::addmm	aten::native_layer_norm
$\downarrow$	$\downarrow$
cuBLAS front-end	$\downarrow$

Fig. 3 : Dispatch chains for library-mediated and framework-native kernels. For library mediated, cuBLAS front-end contributes

\Delta{CT}

for

\mathbb{I}_{lib}=1

Panel of Figure 5 — Fig. 5 : End-to-end latency across dense and MoE LLM workloads. Heatmaps show end-to-end latency across batch sizes and input sequence lengths during prefill ( $m=1$ ) and decode ( $m=10$ ) on H100/H200 systems, where $m$ denotes the number of generated tokens. The decode heatmaps report total latency aggregated over a 10-token decode window. OLMoE-1B/7B does not support SL=8192 context length.

Fig. 6 : Idle fraction across batch size and sequence length on H200 . The idle fraction indicates the portion of latency during which the GPU is not fully executing kernels.

Metric	Llama-3.2-1B	Llama-3.2-3B	OLMoE-1B/7B	Qwen1.5-MoE-A2.7B
Total kernel launches	8,475	15,369	93,053	66,951
Unique kernel names	77	76	222	223
Kernels per token	847.5	1,536.9	9,305.3	6,695.1
Diversity ratio	0.0091	0.0049	0.0024	0.0033
GPU utilization (%)	58.9	67.6	15.5	27.7

TABLE II: Kernel fragmentation for dense vs. MoE models on H100 (BS=4/SL=2048,

m

=10).

GPU	avg	p50	p5	p95
H100	4.707	4.578	4.260	5.396
H200	4.503	4.452	4.177	4.909

TABLE III: Null-kernel

T_{sys}^{floor}

(

\mu

s) measured in isolation.

	Llama-3.2-3B ( $\mu$ s)				OLMoE-1B/7B ( $\mu$ s)
Kernel Family	$p50$	$p95$	$\Delta KT_{fw}$	$\uparrow$	$p50$	$p95$	$\Delta KT_{fw}$	$\uparrow$
$\mathbf{T_{sys}^{floor}}$ (null)	4.75	4.75	–	–	4.73	4.73	–	–
Scan (prefix)	5.07	5.16	0.32	7%	5.12	5.13	0.40	8%
Elem. (unroll)	5.11	5.28	0.36	8%	5.11	5.20	0.38	8%
Elem. (vector)	5.13	5.38	0.38	8%	5.30	5.49	0.58	12%
Reduce	5.30	5.30	0.55	12%	5.07	5.46	0.34	7%

TABLE IV: Per-family launch latency (in

\mu

s) on H100 relative to the floor

T_{\text{sys}}^{floor}

for Llama-3.2-3B and OLMoE-1B/7B (BS=1/SL=512; prefill). Most scan, reduction, and elementwise families remain close to the floor, while GEMM families show the highest

\Delta{KT}_{fw}

. The

T_{sys}^{floor}

row reports the in-context replay floor, which differs slightly from the standalone measurement in Table III (

\approx

0.04

\mu

s) due to process-level CUDA context state.

Panel of Figure 10 — Fig. 7 : Host vs device behavior for GPT‑2 (SL=512) across batch sizes on H200.

Fig. 8 : H200

T_{Orchestration}

and HDBI characterization across dense and MoE workloads. Stacked bars decompose

T_{Orchestration}

into Python, dispatch, CUDA-library, and

T_{sys}

contributions; adjacent bars show

T_{DeviceActive}

; diamonds show HDBI. Prefill uses

m

=1. Decode bars and HDBI points are totals aggregated over the matched

m

=10 decode run. Note: y-axis scales are different between panels; for cross-panel absolute-time comparisons use the HDBI diamonds with the same scale.

Fig. 9 : Eager vs. FlashAttention-2 for Llama-3.2-1B on H200. Bars show end-to-end runtime and

T_{Orchestration}

; lines show GPU utilization and HDBI for BS=1/SL=512 and BS=8/SL=2048.

Fig. 10 : Latency decomposition: H100 vs. H200. Stacked bars show

T_{Orchestration}

(blue) and

T_{DeviceActive}

(green) for each H100/H200 pair. Darker = H100; lighter = H200.

T_{Orchestration}

is consistently shorter on H200, while

T_{DeviceActive}

is comparable or slightly longer. Decode is aggregated latency over 10 output tokens.

Fig. 11 : E2E latency gain (H100

\to

H200) vs. HDBI. Marker shape encodes model

\times

phase; filled is BS=1/SL=512, hollow is BS=4/SL=2048. Host-bound points show largest gains; device-bound points show smaller gains. Decode is aggregate latency over 10 output tokens.

LLM Inference Efficiency

Large language model inference efficiency reduces the time, memory, energy, and cost of generating useful outputs.

See all 350 papers

Aug 9, 2026

Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference

Large Language Models are deployed to multiple types of environments, from internet browsers to edge devices, and WebGPU serves as a modern cross-platform standard. The engines for browser-based LLM inference have proliferated, yet the overhead of WebGPU per-operation dispatch remains poorly characterized. In this work, we introduce a sequential-dispatch measurement method and show that naive single-operation measurements overestimate per-dispatch cost by conflating dispatch with synchronization. Using our method, we measure the per-dispatch cost and show that it is independent of data type used. We show that the dispatch overhead, not kernel quality, is the bottleneck at batch size 1, and isolate the dispatch count as the cause. Therefore, we conclude that at batch size 1, the effective approach to LLM inference optimization in WebGPU is reducing dispatch count. Our findings point to dispatch amortization, in the inference engines and in the WebGPU specification, as a path to practical browser-based inference.

Jędrzej Maczan

1Independent Researcher

LLM Inference Efficiency LLM Inference

Ask PDF

Apr 8, 2026

Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC

Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized. We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement. Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47

\times

and P99 TPOT by up to 3.40

\times

, improving decode throughput by up to 2.1

\times

, and reducing energy per token by up to 48.6

\%

. Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude.

Mohammad Siavashi, Mariano Scazzariello, Gerald Q. Maguire +2

KTH Royal Institute of Technology, Stockholm, Sweden · RISE, Stockholm, Sweden

LLM Serving LLM Inference LLM Inference Efficiency LLM Inference Scheduling

Ask PDF

Aug 22, 2026

LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization

Large language models have become increasingly capable agents for low-level code and kernel optimization, but isolated kernel benchmarks provide only a proxy for the deployment behavior that matters in language-model inference. We identify a benchmark-to-deployment gap: candidate kernels that appear correct and fast in standalone harnesses can exhibit different performance, safety, or phase behavior after integration into a real inference workload. We introduce LLM4LLM, a deployment-aware closed-loop optimization framework that starts from a target inference script, extracts phase-aware optimization tasks, searches with an experience-guided episodic agent, and accepts patches through in-model validation. Across ten language-model inference workloads on A100 and H100 GPUs, LLM4LLM improves end-to-end latency for every evaluated model, achieving 3.91

\times

/6.98

\times

geometric-mean speedups on A100/H100; as supporting kernel-level evidence, it also attains up to 2.745

\times

GeoMean speedup on KernelBench Level 2.

Hui Zeng, Pengfei Yang, Yanxin Chen +2

Xidian University · Zhongguancun Academy · National Key Laboratory of Advanced Communication Networks, Shijiazhuang, Hebei, P. R. China +1

LLM Inference Acceleration GPU Kernel Optimization Large Language Model-Guided Optimization

Ask PDF

Abstract

LLM Inference Efficiency

Explore similar work

Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference

Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC

LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization