cs.DCJul 13, 2026

Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs

Authors: Weijia Han, Lisha Qu

Organizations: University of Washington

Abstract

Reported serving speedups from quantized kernels typically bundle the weight format, the kernel, and the inference runtime into one number. We present an attribution study on four NVIDIA RTX A5000 GPUs, 24 GiB each, on a single host with NVLink-bridged pairs. A matched intermediate stack that keeps the faster runtime without the quantized kernel splits the full speedup into a runtime part and a kernel and quantization part. Under matched greedy decoding the full stack reaches 2.58×2.58\times end to end, with the runtime change accounting for about two thirds of that gain on a logarithmic scale; across three similar model families the kernel and quantization part moves by at most 1.5%. Sharding one instance across all four cards falls well below doubling: a profiler trace attributes about 80% of the per token shortfall to coordination, and an NVLink versus PCIe control on the same hardware shows similar realized bandwidth on both links, pointing away from link bandwidth as the cause. Whether to run one sharded instance or several independent ones depends on the workload and the model, with the ranking reversing on the larger model: the smaller model splits between sharding and multiple instances by workload, while the larger model favors two paired instances on every workload. Quantization extends sustainable concurrent users roughly four times past a reproducible half precision memory cliff. Differences in sampling mode and prompt pool between the two stacks are documented as threats to validity.

Explore similar work

CardsList
Jun 7, 2026cs.DC

APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

W4A4 quantization promises full utilization of INT4 Tensor Cores, yet group dequantization overhead on CUDA Cores has driven existing systems to mixed-precision fallbacks. We present the first systematic study of how intra-SM compute balance governs this bottleneck. Through controlled benchmarks across four GPUs from Ampere and Ada architectures, we identify the Tensor Cores to CUDA Cores throughput ratio (ρρ) as the primary hardware indicator: the W4A4-g128 kernel yields 2.02.0--2.5×2.5\times speedup on RTX3090 (ρ=16ρ=16) yet degrades to 0.430.43--0.47×0.47\times on A100 (ρ=64ρ=64) in compute-bond scenarios, establishing W4A4 viability as platform-dependent rather than universally infeasible. Guided by this finding, we build \textbf{APEX4}, which co-designs pure INT4 GEMM kernels with ρρ-aware granularity adaptation to mitigate the CUDA Cores dequantization bottleneck. APEX4 achieves perplexity within 0.63 of FP16 on LLaMA-2-70B and outperforms W4Ax Atom-g128 by 4.0%--4.4% in zero-shot accuracy. Deployed as a drop-in replacement in unmodified vLLM, it delivers up to 1.66×1.66\times end-to-end speedup on L40S (ρ=8ρ=8), and 1.78×1.78\times on RTX3090 (ρ=16ρ=16), 2.09×2.09\times on A40 (ρ=16ρ=16), while recovering A100 (ρ=64ρ=64) to 1.201.20--1.40×1.40\times via the mixed-granularity mode. Our code is available at https://github.com/APEX4-W4A4/APEX4-W4A4.
Hong Guo, Nianhui Guo, Weixing Wang +3
Tensor ProgramsLLM Inference EfficiencyBalanced Learning
AskPDF
May 7, 2026cs.PF

When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon

KV-cache quantization is framed as a quality--latency trade-off. We show it is \emph{inverted} on Apple Silicon's unified memory: a single fused Metal kernel (sign-randomized FFT ++ per-channel λλ ++ per-group abs-max ++ int4 nibble pack), exposed as a HuggingFace \texttt{Cache} subclass, runs \emph{faster than fp16} across 256256--40964096-token prefixes on Gemma-3 1B (−3-3 to −8%-8\% ms/tok) and at short context on Qwen2.5-1.5B (−0.7-0.7 to −2.6%-2.6\% through 11K), with 3×3\times persistent memory compression and quality preserved (\dPPL=0.000\dPPL = 0.000 Qwen short-prompt; +3.6+3.6 hook \dPPL\dPPL Gemma). The kernel's ∼ ⁣25\sim\!25,ns/vec overhead is below the bandwidth savings from 3×3\times compression. The fused kernel also closes Qwen's 4-bit per-token catastrophe (\dPPL=+7975→+638.6\dPPL = +7975 \to +638.6, 12.5×12.5\times reduction) at 182182,GFLOPS / D=128D{=}128. Supporting findings: \SRFT\SRFT and \SRHT\SRHT are statistically indistinguishable for KV quality (we pick \SRFT\SRFT for mixed-radix and matrix-multiply alignment); a learned-rotation ablation surfaces a regularization role for the fixed random SRFT base (learning R+λR+λ without SRFT lowers calibration MSE 84.9%84.9\% vs 50.3%50.3\% but yields worse PPL); Householder rotations at k=d/2k{=}d/2 reflectors are effectively lossless at d=256d{=}256.
Mohamed Amine Bergach
Key-Value Cache QuantizationSystematic Fp16 DivergenceQuantizationKey-Value CacheCache
AskPDF
Sep 3, 2026cs.LG

Hardware-Aware FP4 FlashAttention-4

Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies dominate once its matrix products shrink. We address this with \emph{Direct-P} for noncausal inference and a causal path that passes the forward quantization directly into backward. Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13×\times the bfloat16 (BF16) forward throughput on an NVIDIA GB200. The causal path reconstructs probabilities from saved quantized queries and keys and uses 8-bit floating-point (FP8) gradient operands, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14×\times. Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.
Robert Hu
Fp8Tensor ProgramsLow-Bit QuantizationProbability
AskPDF