Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers

Jędrzej Maczan

cs.LGFeb 9, 2026

Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers

Authors: Jędrzej Maczan

Organizations: Independent Researcher

Abstract

WebGPU's security-focused design imposes per-operation validation that compounds across the many small dispatches in neural network inference, yet the true cost of this overhead is poorly characterized. We present a systematic characterization of WebGPU dispatch overhead for LLM inference at batch size 1, spanning four GPU vendors (NVIDIA, AMD, Apple, Intel), two native implementations (Dawn, wgpu-native) and three browsers (Chrome, Safari, Firefox), and two model sizes (Qwen2.5-0.5B and 1.5B). Our primary contribution is a sequential-dispatch methodology that reveals naive single-operation benchmarks overestimate dispatch cost by ${\sim}20\times$ . The true per-dispatch cost of WebGPU API overhead alone is 24-36 $μ$ s on Vulkan and 32-71 $μ$ s on Metal, while the total per-operation overhead including Python cost is ${\sim}95$ ~ $μ$ s, which turns out to be a distinction critical for optimization. On Vulkan, kernel fusion improves throughput by 53%, while CUDA fusion provides no benefit, confirming that per-operation overhead is a primary differentiator. LLM inference was tested across three major operating systems (Linux, Windows, macOS). We built $\texttt{torch-webgpu}$ , a PrivateUse1-based out-of-tree PyTorch backend and an FX-to-WebGPU compiler, which on our reference platform achieves 11--12% of CUDA performance. At dtype-matched float32, RTX PRO 2000 achieves 1.4 $\times$ WebGPU's throughput despite ${\sim}6\times$ less compute than RTX 5090. For dispatch overhead, backend choice is the dominant factor, although implementation choice also matters substantially within a backend (2.2 $\times$ for Metal). In terms of dispatch vs kernel compute efficiency, we conclude that at batch=1 with the current dispatch-heavy pipeline, per-operation overhead dominates regardless of kernel quality. All code, benchmarks, and raw data are open source.

Figures & tables

Experiment	Type	Dtype	Configs	Section
End-to-end LLM inference
torch-webgpu	E2E	fp32	1 (RTX 5090/Dawn)	§4
CUDA baselines	E2E	fp16, fp32	2 GPUs, 2 platforms	§4.2
MPS baselines	E2E	fp16, fp32	1 (Apple M2)	§4.2
CPU baselines	E2E	fp32	3 platforms	§4.2
ONNX Runtime (WebGPU)	E2E	fp32	1 (RTX 5090)	§4

Table 1: Classification of experiments by scope and configuration coverage

Qwen2.5-0.5B-Instruct
Backend	Dtype	Tok/s	95% CI	CV	TTFT (ms)	vs CUDA
CUDA (compiled, RTX 5090)	fp16	185.5	[184.2, 186.8]	0.9%	5.4	1.00 $\times$
CUDA (eager, RTX 5090)	fp16	182.9	[182.3, 183.5]	0.4%	5.5	0.99 $\times$
MPS (Apple M2)	fp16	47.8	[47.7, 48.0]	0.9%	20.9	0.26 $\times$
torch-webgpu (fused, RTX 5090)	fp32	21.0	[20.7, 21.4]	4.0%	41.6	0.11 $\times$
CPU (AMD Ryzen, eager)	fp32	13.7	[13.4, 14.0]	3.2%	72.8	0.07 $\times$

Table 2: End-to-end inference performance across backends (10–30 runs per configuration, 50 tokens/run)

Native GPU (end-to-end inference)
Platform	Processor	Accelerator	Tok/s	95% CI	CV	vs WebGPU
Linux (primary)	RTX 5090	CUDA	185.5	[184.2, 186.8]	0.9%	8.8 $\times$
macOS †	Apple M2	MPS	12.9	[12.6, 13.2]	5.5%	0.61 $\times$
Windows 11 (laptop) †	RTX PRO 2000	CUDA	30.1	[29.7, 30.5]	3.3%	1.4 $\times$
CPU (end-to-end inference)
Linux (primary)	AMD Ryzen 9800X3D	CPU	13.7	[13.4, 14.0]	3.2%	0.65 $\times$

Table 3: Cross-platform performance comparison (Qwen2.5-0.5B)

Quantity	Value (ms)	Type	Source
Directly measured
TTFT (fused)	41.6	Measured	End-to-end benchmark
TTFT (unfused)	71.4	Measured	End-to-end benchmark
Per-dispatch cost	0.024	Measured	Sequential dispatch (exp6/exp7)
Well-constrained derived quantity
Per-operation overhead	0.095	Derived a	(71.4 $-$ 41.6) / 312 fewer ops

Table 4: Approximate TTFT overhead accounting (fused torch-webgpu, RTX 5090/Dawn, Qwen2.5-0.5B). Directly measured quantities are shown in bold. Derived estimates have

\sim

30% uncertainty and should be interpreted as a qualitative ordering of bottlenecks. Components are not additive due to GPU/CPU pipelining overlap.

Configuration	Dispatches saved	Tok/s	TTFT (ms)
No fusion (baseline)	—	13.5	71.4
+ Fused RMSNorm (6 $\rightarrow$ 1)	240/fwd	19.4	46.6
+ Fused MLP gate+up+silu (3 $\rightarrow$ 1)	+48/fwd	20.5	43.3
+ Fused K+V projection (2 $\rightarrow$ 1)	+24/fwd	20.6	41.6
Total improvement	312 fewer	+53%	-41%

Table 5: Impact of kernel fusion (controlled progressive experiment ∗ )

Implementation	Single-op ( $\mu$ s)	Sequential ( $\mu$ s)	Backend
Native implementations (dispatch benchmarks)
Dawn (RTX 5090)	496.8	23.8	Vulkan
wgpu (RTX 5090)	35.8	35.8	Vulkan
wgpu (AMD iGPU)	24.8	24.5	Vulkan
wgpu (Apple M2)	48.3	71.1	Metal
Browsers—practical (dispatch benchmarks; E2E via WebLLM in Table 13 )

Table 6: Per-dispatch cost across WebGPU implementations: Single-op vs Sequential measurement

Implementation	Unfused (ms)	Fused (ms)	Speedup	Backend
wgpu (RTX 5090)	0.101	0.072	1.41 $\times$	Vulkan
wgpu (AMD iGPU)	0.106	0.063	1.67 $\times$	Vulkan
wgpu (Apple M2)	2.03	2.13	0.95 $\times$	Metal
Chrome (RTX 5090)	2.11	1.99	1.06 $\times$	Vulkan
Safari (Apple M2)	0.20	0.22	0.91 $\times$	Metal

Table 7: RMSNorm fusion speedup across implementations (6 dispatches

\rightarrow

Operation	Dimensions	Time (ms)	TFLOP/s	% Peak †
MLP up projection	896 $\times$ 896 $\times$ 4864	6.40	1.22	1.2%
MLP down projection	896 $\times$ 4864 $\times$ 896	3.79	2.06	2.0%
Toy matmul	256 $\times$ 256 $\times$ 256	1.10	0.030	$<$ 0.1%

Table 8: WebGPU kernel compute efficiency (wgpu/Vulkan, RTX 5090, 30 runs)

Optimization	Vulkan	Metal	Notes
RMSNorm fusion (6 $\rightarrow$ 1)	✓ 1.4 $\times$	$\times$ 0.95 $\times$	Helps Vulkan only
Tiled MLP (7 $\rightarrow$ 3 disp)	✓ 1.17 $\times$	✓ 2.0 $\times$	Significant on both
Command batching	$\times$ minimal	$\times$ minimal	Sync per token negates benefit

Table 9: Optimization recommendations by target backend

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Operations	Count
Linear (matmul)	Q, K, V, O proj, MLP	169
Multiply	RMSNorm weights, MLP gate	220
Add	Residuals, biases	145
SDPA	Attention per layer ( Dao et al., 2022 ; Dao, 2024 )	24
SiLU	MLP activation	24
RMSNorm components	pow, mean, rsqrt	147

Appendix

Table 10: FX graph operation breakdown (sum = 876 compute ops)

Platform	Backend	Mega (ms)	Multi (ms)	Speedup	$p$ -value	Result
RTX 5090	Vulkan	0.090 $\pm$ 0.03	0.085 $\pm$ 0.01	0.95 $\times$	0.43	Inconclusive
Apple M2	Metal	1.45 $\pm$ 0.32	1.40 $\pm$ 0.02	0.97 $\times$	0.38	Inconclusive

Appendix

Table 11: Mega-kernel vs multi-workgroup: Preliminary comparison at toy scale (256

\times

256, 30 runs). Results are inconclusive for production dimensions.

Dimensions (M $\times$ K $\times$ N)	Workgroups	Mean (ms)	Std (ms)	GFLOP/s
256 $\times$ 256 $\times$ 256	16 $\times$ 16	1.10	0.09	30.4
896 $\times$ 896 $\times$ 4864 (MLP up)	56 $\times$ 304	6.40	0.46	1216
896 $\times$ 4864 $\times$ 896 (MLP down)	56 $\times$ 56	3.79	0.02	2055

Appendix

Table 12: WebGPU matmul at production vs toy dimensions (wgpu/Vulkan, RTX 5090)

Platform	Browser	Model	Decode (tok/s)	Prefill (tok/s)	Backend
Windows 11 (RTX PRO 2000 Blackwell, 8GB)
Windows	Chrome 144	Qwen2.5-0.5B	51.1 $\pm$ 5.9	$\sim$ 650	D3D12
Windows	Chrome 144	Qwen2.5-1.5B	45.7 $\pm$ 6.3	$\sim$ 350	D3D12
Windows	Firefox 147	Qwen2.5-0.5B	9.1 $\pm$ 0.03	$\sim$ 73	D3D12
Windows	Firefox 147	Qwen2.5-1.5B	9.1 $\pm$ 0.03	$\sim$ 55	D3D12
macOS (Apple M2, 16GB unified memory)

Appendix

Table 13: Browser end-to-end LLM inference via WebLLM (q4f16 quantization, decode tok/s from runtime stats ∗ )

Operation	Dimensions ( $d_{\text{in}}\times d_{\text{out}}$ )	$B^{*}$ (computed)	Regime at $B$ =1
Qwen2.5-0.5B (896 hidden, 4864 intermediate)
Attention Q/K/V proj	896 $\times$ 896	119	Overhead-bound
MLP up projection	896 $\times$ 4864	22	Overhead-bound
MLP down projection	4864 $\times$ 896	22	Overhead-bound
Qwen2.5-1.5B (1536 hidden, 8960 intermediate)
Attention Q/K/V proj	1536 $\times$ 1536	40	Overhead-bound

Appendix

Table 14: Dispatch-bound crossover batch size

B^{*}

for representative operations

Platform	Full Readback (ms)	Device Argmax (ms)	Improvement	$p$ -value
wgpu/Vulkan (RTX 5090)	0.42 $\pm$ 0.08	0.12 $\pm$ 0.03	+71% †	0.35
wgpu/Metal (Apple M2)	1.59 $\pm$ 0.12	1.69 $\pm$ 0.15	-7%	0.62

Appendix

Table 15: Device-side argmax: Cross-platform comparison (30 runs)

Kernel optimizations
Optimization	Implementation	Isolated Result	E2E Impact
Parallel softmax	Shared memory, 256 threads	84 $\times$ ( $p<0.001$ )	Bottleneck removed
Tiled matmul	16 $\times$ 16 tiles	2–3 $\times$ ( $p<0.001$ )	$<$ 5% improvement
Overhead reduction attempts (null results)
Command batching	16 dispatches per submit	$\sim$ 0%	No effect ∗
Buffer pooling	Size-class reuse	$\sim$ 0%	No effect

Appendix

Table 16: Optimization results summary (30 runs, RTX 5090)

Metric	CUDA	WebGPU (Vulkan)
Kernel launch/dispatch overhead	7.4 $\pm$ 9.2 $\mu$ s	24–36 $\mu$ s
Overhead ratio	3–5 $\times$ (WebGPU higher)
RMSNorm unfused	21.3 $\mu$ s	—
RMSNorm fused	23.2 $\mu$ s	—
RMSNorm compiled (torch.compile)	20.9 $\mu$ s	—
Fusion speedup	0.92 $\times$ ( no benefit )

Appendix

Table 17: CUDA vs WebGPU: Overhead and fusion comparison (sequential measurement)

Metric	0.5B (measured)	1.5B (measured)	Scaling	Source
Layers	24	28	1.17 $\times$	Model config
Ops/forward (fused)	564	$\sim$ 658 a	1.17 $\times$	Scaled by layers
WebGPU tok/s (fused)	21.0	17.9	0.85 $\times$	Measured (30 runs)
WebGPU tok/s (unfused)	13.5	10.4	0.77 $\times$	Measured (30 runs)
WebGPU TTFT fused (ms)	41.6	51.3	1.23 $\times$	Measured
WebGPU TTFT unfused (ms)	71.4	87.9	1.23 $\times$	Measured

Appendix

Table 18: Model size scaling: 0.5B vs 1.5B, both measured end-to-end (RTX 5090, batch=1 decode, 30 runs)

Platform	Unfused	Tiled	Mega	Tiled Speedup	$p$ -value †
	(7 disp)	(3 disp)	(1 disp)	vs Unfused
wgpu/Vulkan (RTX 5090)	0.72 ms	0.62 ms	0.32 ms	1.17 $\times$	$<$ 0.01
wgpu/Metal (Apple M2)	5.74 ms	2.85 ms	1.48 ms	2.01 $\times$	$<$ 0.001

Appendix

Table 19: Multi-dispatch tiled strategy: Cross-platform comparison (MLP block, 30 runs)

Operation	Total ( $\mu$ s)	Per-dispatch ( $\mu$ s)
Encoder create	644.9	6.4
Pass begin	317.1	3.2
Set pipeline	144.9	1.4
Set bind group	99.2	1.0
Dispatch call	63.9	0.6
Pass end	66.6	0.7

Appendix

Table 20: Per-dispatch timing breakdown (wgpu/Vulkan, 100 dispatches)

LLM Inference Efficiency

Large language model inference efficiency reduces the time, memory, energy, and cost of generating useful outputs.

See all 352 papers

Aug 9, 2026

Dispatch Overhead, Not Kernel Quality: The Performance Bottleneck in Single-Stream WebGPU LLM Inference

Large Language Models are increasingly deployed in resource-constrained, single-stream settings, from internet browsers to edge devices, where requests arrive one at a time at batch size 1 rather than in large batches. In this regime, where WebGPU serves as a modern cross-platform standard, per-operation dispatch overhead, rather than kernel compute, can dominate inference cost, yet it remains poorly characterized. In this work, we introduce a sequential-dispatch measurement method and show that naive single-operation measurements overestimate per-dispatch cost by conflating dispatch with synchronization. Using our method, we measure the per-dispatch cost and show that it is independent of data type used. Through a controlled fusion experiment, we show that the dispatch overhead, not kernel quality, is the bottleneck at batch size 1, and isolate the dispatch count as the cause. We conclude that at batch size 1, the effective approach to LLM inference optimization in WebGPU is reducing dispatch count. Our findings point to dispatch amortization, in the inference engines and in the WebGPU specification, as a path to practical browser-based inference.

Jędrzej Maczan

Independent Researcher

LLM Inference Efficiency LLM Inference

Ask PDF

Oct 12, 2026

GPU Goldmines: Tunable WebGPU Kernels for Edge LLM Inference

As demand for LLM inference grows and consumer GPUs grow more capable, edge deployment is becoming increasingly attractive. WebGPU is compelling in this setting because it offers cross-platform portability and browser accessibility. However, these same strengths also make high performance difficult to achieve, as an implementation that performs well in one context may not perform well in another. To address this, we present GPU Goldmines, a framework for specifying, executing, and tuning WebGPU kernels. We instantiate GPU Goldmines with kernels for key LLM building blocks: matrix multiplication, matrix-vector multiplication, and flash attention. We characterize performance across diverse devices, shapes, and browsers, and use the resulting database to accelerate subsequent autotuning and configuration selection. On shapes drawn from Llama and Qwen models, our matrix multiplication kernels outperform ONNX Runtime and llama.cpp WebGPU kernels by 7.0x and 1.41x geometric mean, respectively, and deliver end-to-end improvements of 1.48x in the prefill stage of llama.cpp for chat workloads.

Rithik Sharma, Reese Levine, Zheyuan Chen +3

UC Santa Cruz · EPFL · Microsoft Research

GPU Kernel Optimization LLM Inference Acceleration On-Device Language Model Inference

Ask PDF

May 20, 2026

Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU

Running language models in the browser presents a unique opportunity to build efficient, private, and portable AI applications, but requires contending with constrained memory availability and heterogeneous hardware targets. To realize this opportunity, we present Llamas on the Web (LlamaWeb), a WebGPU backend for llama

.

cpp that enables memory-efficient and performance-portable LLM inference across a wide range of model weight formats in the browser. Our design significantly reduces memory overhead through static memory planning and efficient model loading, addresses cross-device variability through a tunable kernel library, and introduces templated GPU kernels that support performant implementations of numerous quantization formats, enabling broad model support and extensibility to new formats. We evaluate LlamaWeb on 16 devices from 8 vendors, collecting data from 10 language models and four model weight formats. We compare LlamaWeb against existing browser-based LLM frameworks and find that LlamaWeb requires 29-33% less memory across several combinations of device, browser, and operating system. We also evaluate LlamaWeb's performance against these frameworks and find that it increases decode throughput by 45-69% across four GPUs from separate vendors. In addition, we compare LlamaWeb's performance against other llama

.

cpp backends, where it is competitive with and even beats vendor-specific backend performance on some devices.

Reese Levine, Rithik Sharma, Nikhil Jain +5

UC Santa Cruz · Microsoft Research

GPU Acceleration LLM Inference Memory-Efficient Inference LLM Inference Acceleration

Ask PDF

Abstract

LLM Inference Efficiency

Explore similar work

Dispatch Overhead, Not Kernel Quality: The Performance Bottleneck in Single-Stream WebGPU LLM Inference

GPU Goldmines: Tunable WebGPU Kernels for Edge LLM Inference

Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU