Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers

Jędrzej Maczan

cs.LGFeb 9, 2026

Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers

Authors: Jędrzej Maczan

Organizations: Independent Researcher

Abstract

WebGPU's security-focused design imposes per-operation validation that compounds across the many small dispatches in neural network inference, yet the true cost of this overhead is poorly characterized. We present a systematic characterization of WebGPU dispatch overhead for LLM inference at batch size 1, spanning four GPU vendors (NVIDIA, AMD, Apple, Intel), two native implementations (Dawn, wgpu-native) and three browsers (Chrome, Safari, Firefox), and two model sizes (Qwen2.5-0.5B and 1.5B). Our primary contribution is a sequential-dispatch methodology that reveals naive single-operation benchmarks overestimate dispatch cost by ${\sim}20\times$ . The true per-dispatch cost of WebGPU API overhead alone is 24-36 $μ$ s on Vulkan and 32-71 $μ$ s on Metal, while the total per-operation overhead including Python cost is ${\sim}95$ ~ $μ$ s, which turns out to be a distinction critical for optimization. On Vulkan, kernel fusion improves throughput by 53%, while CUDA fusion provides no benefit, confirming that per-operation overhead is a primary differentiator. LLM inference was tested across three major operating systems (Linux, Windows, macOS). We built $\texttt{torch-webgpu}$ , a PrivateUse1-based out-of-tree PyTorch backend and an FX-to-WebGPU compiler, which on our reference platform achieves 11--12% of CUDA performance. At dtype-matched float32, RTX PRO 2000 achieves 1.4 $\times$ WebGPU's throughput despite ${\sim}6\times$ less compute than RTX 5090. For dispatch overhead, backend choice is the dominant factor, although implementation choice also matters substantially within a backend (2.2 $\times$ for Metal). In terms of dispatch vs kernel compute efficiency, we conclude that at batch=1 with the current dispatch-heavy pipeline, per-operation overhead dominates regardless of kernel quality. All code, benchmarks, and raw data are open source.

Figures & tables

Experiment	Type	Dtype	Configs	Section
End-to-end LLM inference
torch-webgpu	E2E	fp32	1 (RTX 5090/Dawn)	§4
CUDA baselines	E2E	fp16, fp32	2 GPUs, 2 platforms	§4.2
MPS baselines	E2E	fp16, fp32	1 (Apple M2)	§4.2
CPU baselines	E2E	fp32	3 platforms	§4.2
ONNX Runtime (WebGPU)	E2E	fp32	1 (RTX 5090)	§4

Table 1: Classification of experiments by scope and configuration coverage

Qwen2.5-0.5B-Instruct
Backend	Dtype	Tok/s	95% CI	CV	TTFT (ms)	vs CUDA
CUDA (compiled, RTX 5090)	fp16	185.5	[184.2, 186.8]	0.9%	5.4	1.00 $\times$
CUDA (eager, RTX 5090)	fp16	182.9	[182.3, 183.5]	0.4%	5.5	0.99 $\times$
MPS (Apple M2)	fp16	47.8	[47.7, 48.0]	0.9%	20.9	0.26 $\times$
torch-webgpu (fused, RTX 5090)	fp32	21.0	[20.7, 21.4]	4.0%	41.6	0.11 $\times$
CPU (AMD Ryzen, eager)	fp32	13.7	[13.4, 14.0]	3.2%	72.8	0.07 $\times$

Table 2: End-to-end inference performance across backends (10–30 runs per configuration, 50 tokens/run)

Native GPU (end-to-end inference)
Platform	Processor	Accelerator	Tok/s	95% CI	CV	vs WebGPU
Linux (primary)	RTX 5090	CUDA	185.5	[184.2, 186.8]	0.9%	8.8 $\times$
macOS †	Apple M2	MPS	12.9	[12.6, 13.2]	5.5%	0.61 $\times$
Windows 11 (laptop) †	RTX PRO 2000	CUDA	30.1	[29.7, 30.5]	3.3%	1.4 $\times$
CPU (end-to-end inference)
Linux (primary)	AMD Ryzen 9800X3D	CPU	13.7	[13.4, 14.0]	3.2%	0.65 $\times$

Table 3: Cross-platform performance comparison (Qwen2.5-0.5B)

Quantity	Value (ms)	Type	Source
Directly measured
TTFT (fused)	41.6	Measured	End-to-end benchmark
TTFT (unfused)	71.4	Measured	End-to-end benchmark
Per-dispatch cost	0.024	Measured	Sequential dispatch (exp6/exp7)
Well-constrained derived quantity
Per-operation overhead	0.095	Derived a	(71.4 $-$ 41.6) / 312 fewer ops

Table 4: Approximate TTFT overhead accounting (fused torch-webgpu, RTX 5090/Dawn, Qwen2.5-0.5B). Directly measured quantities are shown in bold. Derived estimates have

\sim

30% uncertainty and should be interpreted as a qualitative ordering of bottlenecks. Components are not additive due to GPU/CPU pipelining overlap.

Configuration	Dispatches saved	Tok/s	TTFT (ms)
No fusion (baseline)	—	13.5	71.4
+ Fused RMSNorm (6 $\rightarrow$ 1)	240/fwd	19.4	46.6
+ Fused MLP gate+up+silu (3 $\rightarrow$ 1)	+48/fwd	20.5	43.3
+ Fused K+V projection (2 $\rightarrow$ 1)	+24/fwd	20.6	41.6
Total improvement	312 fewer	+53%	-41%

Table 5: Impact of kernel fusion (controlled progressive experiment ∗ )

Implementation	Single-op ( $\mu$ s)	Sequential ( $\mu$ s)	Backend
Native implementations (dispatch benchmarks)
Dawn (RTX 5090)	496.8	23.8	Vulkan
wgpu (RTX 5090)	35.8	35.8	Vulkan
wgpu (AMD iGPU)	24.8	24.5	Vulkan
wgpu (Apple M2)	48.3	71.1	Metal
Browsers—practical (dispatch benchmarks; E2E via WebLLM in Table 13 )

Table 6: Per-dispatch cost across WebGPU implementations: Single-op vs Sequential measurement

Implementation	Unfused (ms)	Fused (ms)	Speedup	Backend
wgpu (RTX 5090)	0.101	0.072	1.41 $\times$	Vulkan
wgpu (AMD iGPU)	0.106	0.063	1.67 $\times$	Vulkan
wgpu (Apple M2)	2.03	2.13	0.95 $\times$	Metal
Chrome (RTX 5090)	2.11	1.99	1.06 $\times$	Vulkan
Safari (Apple M2)	0.20	0.22	0.91 $\times$	Metal

Table 7: RMSNorm fusion speedup across implementations (6 dispatches

\rightarrow

Operation	Dimensions	Time (ms)	TFLOP/s	% Peak †
MLP up projection	896 $\times$ 896 $\times$ 4864	6.40	1.22	1.2%
MLP down projection	896 $\times$ 4864 $\times$ 896	3.79	2.06	2.0%
Toy matmul	256 $\times$ 256 $\times$ 256	1.10	0.030	$<$ 0.1%

Table 8: WebGPU kernel compute efficiency (wgpu/Vulkan, RTX 5090, 30 runs)

Optimization	Vulkan	Metal	Notes
RMSNorm fusion (6 $\rightarrow$ 1)	✓ 1.4 $\times$	$\times$ 0.95 $\times$	Helps Vulkan only
Tiled MLP (7 $\rightarrow$ 3 disp)	✓ 1.17 $\times$	✓ 2.0 $\times$	Significant on both
Command batching	$\times$ minimal	$\times$ minimal	Sync per token negates benefit

Table 9: Optimization recommendations by target backend

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Operations	Count
Linear (matmul)	Q, K, V, O proj, MLP	169
Multiply	RMSNorm weights, MLP gate	220
Add	Residuals, biases	145
SDPA	Attention per layer ( Dao et al., 2022 ; Dao, 2024 )	24
SiLU	MLP activation	24
RMSNorm components	pow, mean, rsqrt	147

Appendix

Table 10: FX graph operation breakdown (sum = 876 compute ops)

Platform	Backend	Mega (ms)	Multi (ms)	Speedup	$p$ -value	Result
RTX 5090	Vulkan	0.090 $\pm$ 0.03	0.085 $\pm$ 0.01	0.95 $\times$	0.43	Inconclusive
Apple M2	Metal	1.45 $\pm$ 0.32	1.40 $\pm$ 0.02	0.97 $\times$	0.38	Inconclusive

Appendix

Table 11: Mega-kernel vs multi-workgroup: Preliminary comparison at toy scale (256

\times

256, 30 runs). Results are inconclusive for production dimensions.

Dimensions (M $\times$ K $\times$ N)	Workgroups	Mean (ms)	Std (ms)	GFLOP/s
256 $\times$ 256 $\times$ 256	16 $\times$ 16	1.10	0.09	30.4
896 $\times$ 896 $\times$ 4864 (MLP up)	56 $\times$ 304	6.40	0.46	1216
896 $\times$ 4864 $\times$ 896 (MLP down)	56 $\times$ 56	3.79	0.02	2055

Appendix

Table 12: WebGPU matmul at production vs toy dimensions (wgpu/Vulkan, RTX 5090)

Platform	Browser	Model	Decode (tok/s)	Prefill (tok/s)	Backend
Windows 11 (RTX PRO 2000 Blackwell, 8GB)
Windows	Chrome 144	Qwen2.5-0.5B	51.1 $\pm$ 5.9	$\sim$ 650	D3D12
Windows	Chrome 144	Qwen2.5-1.5B	45.7 $\pm$ 6.3	$\sim$ 350	D3D12
Windows	Firefox 147	Qwen2.5-0.5B	9.1 $\pm$ 0.03	$\sim$ 73	D3D12
Windows	Firefox 147	Qwen2.5-1.5B	9.1 $\pm$ 0.03	$\sim$ 55	D3D12
macOS (Apple M2, 16GB unified memory)

Appendix

Table 13: Browser end-to-end LLM inference via WebLLM (q4f16 quantization, decode tok/s from runtime stats ∗ )

Operation	Dimensions ( $d_{\text{in}}\times d_{\text{out}}$ )	$B^{*}$ (computed)	Regime at $B$ =1
Qwen2.5-0.5B (896 hidden, 4864 intermediate)
Attention Q/K/V proj	896 $\times$ 896	119	Overhead-bound
MLP up projection	896 $\times$ 4864	22	Overhead-bound
MLP down projection	4864 $\times$ 896	22	Overhead-bound
Qwen2.5-1.5B (1536 hidden, 8960 intermediate)
Attention Q/K/V proj	1536 $\times$ 1536	40	Overhead-bound

Appendix

Table 14: Dispatch-bound crossover batch size

B^{*}

for representative operations

Platform	Full Readback (ms)	Device Argmax (ms)	Improvement	$p$ -value
wgpu/Vulkan (RTX 5090)	0.42 $\pm$ 0.08	0.12 $\pm$ 0.03	+71% †	0.35
wgpu/Metal (Apple M2)	1.59 $\pm$ 0.12	1.69 $\pm$ 0.15	-7%	0.62

Appendix

Table 15: Device-side argmax: Cross-platform comparison (30 runs)

Kernel optimizations
Optimization	Implementation	Isolated Result	E2E Impact
Parallel softmax	Shared memory, 256 threads	84 $\times$ ( $p<0.001$ )	Bottleneck removed
Tiled matmul	16 $\times$ 16 tiles	2–3 $\times$ ( $p<0.001$ )	$<$ 5% improvement
Overhead reduction attempts (null results)
Command batching	16 dispatches per submit	$\sim$ 0%	No effect ∗
Buffer pooling	Size-class reuse	$\sim$ 0%	No effect

Appendix

Table 16: Optimization results summary (30 runs, RTX 5090)

Metric	CUDA	WebGPU (Vulkan)
Kernel launch/dispatch overhead	7.4 $\pm$ 9.2 $\mu$ s	24–36 $\mu$ s
Overhead ratio	3–5 $\times$ (WebGPU higher)
RMSNorm unfused	21.3 $\mu$ s	—
RMSNorm fused	23.2 $\mu$ s	—
RMSNorm compiled (torch.compile)	20.9 $\mu$ s	—
Fusion speedup	0.92 $\times$ ( no benefit )

Appendix

Table 17: CUDA vs WebGPU: Overhead and fusion comparison (sequential measurement)

Metric	0.5B (measured)	1.5B (measured)	Scaling	Source
Layers	24	28	1.17 $\times$	Model config
Ops/forward (fused)	564	$\sim$ 658 a	1.17 $\times$	Scaled by layers
WebGPU tok/s (fused)	21.0	17.9	0.85 $\times$	Measured (30 runs)
WebGPU tok/s (unfused)	13.5	10.4	0.77 $\times$	Measured (30 runs)
WebGPU TTFT fused (ms)	41.6	51.3	1.23 $\times$	Measured
WebGPU TTFT unfused (ms)	71.4	87.9	1.23 $\times$	Measured

Appendix

Table 18: Model size scaling: 0.5B vs 1.5B, both measured end-to-end (RTX 5090, batch=1 decode, 30 runs)

Platform	Unfused	Tiled	Mega	Tiled Speedup	$p$ -value †
	(7 disp)	(3 disp)	(1 disp)	vs Unfused
wgpu/Vulkan (RTX 5090)	0.72 ms	0.62 ms	0.32 ms	1.17 $\times$	$<$ 0.01
wgpu/Metal (Apple M2)	5.74 ms	2.85 ms	1.48 ms	2.01 $\times$	$<$ 0.001

Appendix

Table 19: Multi-dispatch tiled strategy: Cross-platform comparison (MLP block, 30 runs)

Operation	Total ( $\mu$ s)	Per-dispatch ( $\mu$ s)
Encoder create	644.9	6.4
Pass begin	317.1	3.2
Set pipeline	144.9	1.4
Set bind group	99.2	1.0
Dispatch call	63.9	0.6
Pass end	66.6	0.7

Appendix

Table 20: Per-dispatch timing breakdown (wgpu/Vulkan, 100 dispatches)

LLM Inference Efficiency

Large language model inference efficiency reduces the time, memory, energy, and cost of generating useful outputs.

See all 352 papers

Dispatch Overhead, Not Kernel Quality: The Performance Bottleneck in Single-Stream WebGPU LLM Inference
Aug 9, 2026Jędrzej MaczanLLM Inference Efficiency LLM Inference
Independent Researcher
GPU Goldmines: Tunable WebGPU Kernels for Edge LLM Inference
Oct 12, 2026Rithik Sharma, Reese Levine, Zheyuan Chen +3GPU Kernel Optimization LLM Inference Acceleration
UC Santa Cruz · EPFL · Microsoft Research
Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU
May 20, 2026Reese Levine, Rithik Sharma, Nikhil Jain +5GPU Acceleration LLM Inference
UC Santa Cruz · Microsoft Research

Aug 9, 2026J/K move · Enter open · S save

Dispatch Overhead, Not Kernel Quality: The Performance Bottleneck in Single-Stream WebGPU LLM Inference

Jędrzej Maczan

Independent Researcher

LLM Inference Efficiency LLM Inference

Open paper AskPDF arXiv

Large Language Models are increasingly deployed in resource-constrained, single-stream settings, from internet browsers to edge devices, where requests arrive one at a time at batch size 1 rather than in large batches. In this regime, where WebGPU serves as a modern cross-platform standard, per-operation dispatch overhead, rather than kernel compute, can dominate inference cost, yet it remains poorly characterized. In this work, we introduce a sequential-dispatch measurement method and show that naive single-operation measurements overestimate per-dispatch cost by conflating dispatch with synchronization. Using our method, we measure the per-dispatch cost and show that it is independent of data type used. Through a controlled fusion experiment, we show that the dispatch overhead, not kernel quality, is the bottleneck at batch size 1, and isolate the dispatch count as the cause. We conclude that at batch size 1, the effective approach to LLM inference optimization in WebGPU is reducing dispatch count. Our findings point to dispatch amortization, in the inference engines and in the WebGPU specification, as a path to practical browser-based inference.

Abstract

LLM Inference Efficiency

Explore similar work

Dispatch Overhead, Not Kernel Quality: The Performance Bottleneck in Single-Stream WebGPU LLM Inference

GPU Goldmines: Tunable WebGPU Kernels for Edge LLM Inference

Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU