FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper

Wei Wang

cs.AIOct 16, 2026

FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper

Authors: Wei Wang

Abstract

Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads. We present FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs. It applies CODA's GEMM-plus-epilogue approach to inference, adding direct KV-cache placement, folding the fixed RMSNorm gain into weights, writing FP8 output directly with delayed scaling, and supporting prefill without padding. Decode uses INT4 projection weights and an INT4 KV-cache copy. Weights and keys follow Tensor Core fragment layouts for direct dequantization into MMA operands without full intermediate tiles. A block-diagonal Hadamard transform reduces cache quantization error from outlier channels. On one H100, prefill is up to 1.48x faster than compiled PyTorch FP8 and reaches 1.29 PFLOP/s. For gate/up projection with SwiGLU at the 8B shape, our BF16 path is 1.13x faster than CODA. Our FP8 path is 2.02-2.18x faster than the evaluated BF16 CODA path. Across 8B and 70B shapes at batches 1-32, INT4 decode is 1.3-2.4x faster than the faster of PyTorch's BF16 and INT4 blocks at each point: 2.1-3.3x faster than compiled, CUDA-graphed BF16 and 1.3-6.3x faster than the same block using PyTorch's INT4 projection kernel, which targets a single query row and loses to BF16 at batch 32. It is also 31-51% faster than our FP8 path. Complete Llama-3.1-8B decode takes 2.8 ms per step at batch one, versus 8.1 ms for the compiled transformers implementation. Decoding every WikiText-2 token through the INT4 weight path raises perplexity from 6.24 to 7.23 with round-to-nearest weights or 6.99 with GPTQ in the same format; the rotated INT4 cache adds 0.20. The kernels match a quantized reference computation within 0.001 perplexity.

Figures & tables

Shape	$H$	$I$	$H_{q}$	$H_{kv}$	$D$	Projection weights
1B	2048	8192	32	8	64	60.82 M
8B	4096	14336	32	8	128	218.10 M
70B	8192	28672	64	8	128	855.64 M
405B	16384	53248	128	8	128	3187.67 M

Table 1: Llama-style layer shapes evaluated in this paper.

Panel of Figure 1 — Figure 1: Prefill at batch one with FP8 projections and BF16 attention. Left: speedup over compiled, CUDA-graphed PyTorch FP8. Right: effective block throughput. Each point shows the measured median.

	Latency ( $\mu$ s)			Our speedup
Shape	Compiled PyTorch	CODA	FuTransformer (ours)	vs. PyTorch	vs. CODA
A. Precision matched: all implementations use BF16
8B	726	690	613	1.18 $\times$	1.13 $\times$
405B	10089	9750	9442	1.07 $\times$	1.03 $\times$
B. Cross precision: baselines use BF16; FuTransformer uses E4M3
8B	726	690	316	2.30 $\times$	2.18 $\times$

Table 2: Gate/up projection followed by SwiGLU at

M=2048

on one H100. Latency is in

\mu

s; lower is better. Panel A compares BF16 implementations at matched precision. Panel B compares our E4M3 path with BF16 baselines, so the precisions differ. 8B:

K=4096

2I=28672

. 405B:

K=16384

2I=106496

Shape, batch, context	eager	compiled	INT4 block	BF16 cache	INT4 cache	compiled	INT4 block
	PyTorch, BF16 cache			FuTransformer INT4 weights		speedup vs.
8B, $B=1$ , 1024	1015	172	106	77.8	78.0	2.2 $\times$	1.4 $\times$
8B, $B=8$ , 1024	1019	214	203	76.5	74.1	2.9 $\times$	2.7 $\times$
8B, $B=32$ , 4096	1118	382	717	265.5	180.9	2.1 $\times$	4.0 $\times$
70B, $B=1$ , 1024	1399	591	270	205.3	206.1	2.9 $\times$	1.3 $\times$
70B, $B=8$ , 1024	1405	643	618	196.6	197.4	3.3 $\times$	3.1 $\times$

Table 3: Full decode-layer latency on one H100, in

\mu

s. All PyTorch baselines use a BF16 cache. The INT4 baseline uses the same block as compiled PyTorch, with its four projections replaced by torch._weight_int4pack_mm . Both use the same compilation and CUDA-graph setup. FuTransformer uses INT4 weights with either a BF16 cache, matching PyTorch’s precision, or its INT4 cache copy. Speedups refer to the complete FuTransformer INT4 block.

$B$	compiled	INT4 block	FuTransformer INT4
	PyTorch
1	170	106	77.0
8	215	202	74.6
16	229	318	98.5
32	251	585	118.8

Table 4: Decode-layer latency as batch size increases, with context fixed at 1024 and the 8B shape. Latency is in

\mu

s. The PyTorch INT4 baseline has the same block structure as in Table 3 .

	BF16 cache			E4M3 cache			INT4 cache
Shape, batch, context	BF16	FP8	INT4	BF16	FP8	INT4	BF16	FP8	INT4
8B, $B=1$ , 1024	165.2	122.1	77.0	167.2	124.4	78.0	166.9	122.4	77.0
8B, $B=8$ , 1024	174.8	128.4	76.4	176.1	128.6	78.3	174.0	126.2	75.4
8B, $B=32$ , 4096	364.6	324.4	266.2	306.0	269.9	205.5	275.3	236.8	179.8
70B, $B=1$ , 1024	588.7	334.4	205.0	590.6	335.6	206.2	603.3	334.4	206.8
70B, $B=8$ , 1024	606.6	399.9	196.1	607.0	398.8	199.8	600.8	398.2	195.2

Table 5: Decode-layer latency within FuTransformer under CUDA graph replay, in

\mu

s. Column groups specify the KV-cache format, and columns within each group specify the projection-weight format. INT4 cache results include rotation. The three main decode paths are BF16/BF16, E4M3/E4M3 and INT4/INT4.

Implementation	$B=1$ , 1024	$B=8$ , 1024	$B=32$ , 4096	$B=1$	$B=8$	$B=32$	ms	tokens/s
	Decode, ms per step			Decode, tokens/s			Prefill 2048
PyTorch eager, BF16	22.77	22.30	103.74	44	359	309	67.9	30 147
PyTorch compiled, BF16	8.08	11.08	59.35	124	722	539	52.1	39 318
FuTransformer BF16 / BF16	5.51	5.77	11.71	182	1 386	2 732	46.4	44 153
FuTransformer FP8 / BF16	4.19	4.33	10.56	239	1 849	3 030	26.6	77 010
FuTransformer INT4 / BF16	2.70	2.66	8.76	371	3 004	3 654	46.5	44 067

Table 6: Complete Llama-3.1-8B on one H100: latency and throughput per decode step, and prefill of a 2048-token prompt. Each FuTransformer row lists the projection-weight format / KV-cache format used by its decode blocks.

	8B			70B
Variant	$B=1$	$B=8$	$B=32$ , 4096	$B=1$
FuTransformer as described	77.3	74.8	178.3	208.1
K in natural order, shared-memory transpose	77.9	76.5	207.0	207.5
and nibbles unpacked arithmetically	78.1	78.2	223.0	208.8
Cache row quantized by a separate kernel	79.4	76.7	179.4	209.5
Weight nibbles converted by a bias MMA	84.2	89.8	179.8	231.4

Table 7: Design ablation of the INT4 decode block using INT4 weights and an INT4 cache. Latency is in

\mu

s. Each variant replaces a design element with its earlier version.

	Normal input		Outlier mixture
Configuration	1B	8B	1B	8B
PyTorch BF16	4.13	4.12	4.20	4.18
FuTransformer BF16	3.46	3.43	3.54	3.49
PyTorch FP8, row-wise scale	39.6	39.5	39.1	39.1
FuTransformer FP8, delayed tensor scale	40.3	40.3	39.8	39.8

Table 8: Absolute RMSE of the block output compared with an FP64 reference at

S=512

. Values are in units of

10^{-3}

Configuration	8B	1B
E4M3 cache	0.9	0.7
INT4 cache	3.2	2.2
INT4 cache, channel outliers, no rotation	22.2	29.9
INT4 cache, channel outliers, Hadamard rotation	7.4	12.9
E4M3 weights	3.4–3.5	3.4–3.6
INT4 weights, group size 128	9.8–10.2	9.9–10.5

Table 9: Relative RMS output error for one decode layer, in %. Cache results use a PyTorch BF16 reference. Weight results isolate weight quantization while keeping the cache unchanged. All inputs are synthetic.

Decode-block weights	Perplexity	vs. BF16
PyTorch BF16 (reference)	6.240
FuTransformer BF16 kernels	6.240	+0.000
FuTransformer INT4, round-to-nearest	7.226	+0.986
FuTransformer INT4, round-to-nearest, clipped scales	7.393	+1.153
FuTransformer INT4, GPTQ	6.986	+0.746

Table 10: WikiText-2 perplexity for Llama-3.1-8B. Every token runs through the fused decode blocks with a BF16 KV cache. The test uses 141 windows of 2048 tokens.

		INT4 cache mirror
Decode-block weights	BF16 cache	rotated	unrotated
BF16	6.240	6.387	6.777
INT4, round-to-nearest	7.226	7.458	8.161
INT4, GPTQ	6.986	7.182	7.762

Table 11: WikiText-2 perplexity for Llama-3.1-8B using the fused decode blocks and an INT4 KV-cache copy, with and without Hadamard rotation. The test uses 141 windows of 2048 tokens. The BF16-cache column repeats Table 10 .

	Independent		Cascaded		BF16
Layer	RTN	GPTQ	RTN	GPTQ	kernels
0	13.0	6.1	15.6	7.6	0.28
1	12.6	7.2	20.9	13.1	0.39
2	–	–	30.7	21.4	–
4	–	–	36.5	28.4	–
8	–	–	40.1	29.3	–

Table 12: Relative RMS error of each layer’s decode output, in %, using real Llama-3.1-8B weights and activations. Independent: each layer receives the BF16 model’s input to that layer; results are averaged over four decode tokens after a 512-token prefill. Cascaded: the INT4 stack decodes 16 windows using its own residual stream and KV cache; results cover positions 1024–2047. RTN means round-to-nearest. The last column compares the BF16 kernels with an FP32 reference.

Prefill / decode weights	BF16 cache	INT4 cache
BF16 / BF16	5.595	5.728
BF16 / E4M3	5.616	–
BF16 / INT4 round-to-nearest	6.158	–
BF16 / INT4 GPTQ	5.986	6.120
E4M3 / BF16	5.599	5.731
E4M3 / E4M3	5.621	5.745

Table 13: WikiText-2 perplexity for Llama-3.1-8B using the serving workflow. The first 1024 tokens use prefill, and the remaining 1023 are decoded one at a time. Only decoded positions are scored, over 141 windows. Prefill does not support INT4. A dash indicates an unmeasured combination.

Configuration	HellaSwag	ARC-Challenge
BF16	0.793	0.552
INT4 GPTQ weights, BF16 cache	0.772	0.503
and the rotated INT4 cache	0.768	0.476

Table 14: Zero-shot accuracy ( acc_norm ) of Llama-3.1-8B with the decode path’s quantization. The evaluation model uses the same numerical values as the kernels. Standard errors are 0.004 for HellaSwag and 0.015 for ARC-Challenge.

Length distribution	Actual tokens	Padding ratio	Speedup
All equal	16384	1.00 $\times$	1.00 $\times$
$U(0.7,1.0)\cdot S_{\max}$	14097	1.16 $\times$	1.15 $\times$
$U(0.2,1.0)\cdot S_{\max}$	10448	1.57 $\times$	1.57 $\times$
Lognormal, median $S_{\max}/3$	8527	1.92 $\times$	1.85 $\times$
One at $S_{\max}$ , seven at $S_{\max}/16$	2944	5.57 $\times$	5.07 $\times$

Table 15: Packed and padded prefill compared within the same FuTransformer implementation.

Efficient Transformer Inference

Efficient Transformer Inference studies methods and systems that reduce the latency, memory, and energy needed to run Transformer models.

See all 119 papers

Jun 12, 2026

Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0

Post-training INT8 (W8A8) quantization of diffusion transformers is widely deployed as a speed optimization, yet on consumer Ampere GPUs it is frequently slower than the FP8 and NF4 alternatives it is meant to beat. We trace this to a software artifact: the production "INT8" forward quantizes weights and activations only to immediately dequantize them back to bf16 and run a bf16 matrix multiply, never engaging the GPU's INT8 tensor cores, so the hardware's compute advantage is left entirely unrealized. We close this gap with a single fused Triton INT8 GEMM (int8xint8->int32 on Ampere tensor cores, with per-token x per-channel dequantization and bias folded into the epilogue, autotuned per GEMM shape) dropped into the Ideogram 4.0 diffusion transformer's linear layers in place of the dequantize-to-bf16 path. In the kernel, the int8xint8->int32 accumulation is bit-exact against torch._int_mm and the dequantized output matches the reference at cosine similarity 1.0 with no NaNs, running 2.8-4.2x faster than bf16 per GEMM. End to end it delivers a ~1.1x (~9-10%) speedup at 768px, and at 1024px it generates an image in 156.5 s on a single RTX 3090, faster than the single-card NF4 (164.5 s) and FP8 (172.9 s) baselines, at no measurable quality cost on these point estimates (PickScore/CLIPScore). INT8 thus goes from the slowest variant to the fastest, and 1024px becomes single-GPU feasible. The primary speed criterion (beat FP8, by ~9.5%) is comfortably met; the NF4 margin (~4.9%, single-run n=4) is within run-to-run variance we did not quantify and is best read as consistent with meeting the stretch target. We close with an honest deployment map: the win is specific to consumer Ampere, and on A100 and B200 the same kernel loses to those cards' fast native bf16/FP8 paths.

Ali Asaria, Tony Salomone, Deep Gandhi

Transformer Lab

GPU Kernel Optimization Diffusion Transformer Diffusion Model Quantization Post-Training Quantization Diffusion Model Inference Acceleration

Ask PDF

Jun 5, 2025

FPTQuant: Function-Preserving Transforms for LLM Quantization

Large language models (LLMs) require substantial compute, and thus energy, at inference time. While quantizing weights and activations is effective at improving efficiency, naive quantization of LLMs can significantly degrade performance due to large magnitude outliers. This paper describes FPTQuant, which introduces three novel, lightweight, and expressive function-preserving transforms (FPTs) to facilitate quantization of transformers: (1) a mergeable pre-RoPE transform for queries and keys, (2) a mergeable transform for values, and (3) a cheap, dynamic per-token scaling transform. By leveraging the equivariances and independencies inherent to canonical transformer operation, we designed these FPTs to maintain the model's function while shaping the intermediate activation distributions to be more quantization friendly. FPTQuant requires no custom kernels and adds virtually no overhead during inference. The FPTs are trained both locally to reduce outliers, and end-to-end such that the outputs of the quantized and full-precision models match. FPTQuant enables static INT4 quantization with minimal overhead and shows SOTA speed-up of up to 3.9X over FP. Empirically, FPTQuant has an excellent accuracy-speed trade-off -- it is performing on par or exceeding most prior work and only shows slightly lower accuracy compared to a method that is up to 29% slower.

Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1

Qualcomm AI Research, Amsterdam, Netherlands

LLM Quantization LLM Inference Acceleration Activation Quantization Low-Bit Quantization

Ask PDF

Sep 4, 2026

Format-Aware Fusion for Fast FP4 Pretraining

Four-bit floating-point (FP4) Tensor Cores accelerate matrix multiplication, but scale computation, operand packing, layout construction, and saved backward state can erase the gain. We present format-aware fusion, which co-designs each quantization producer with its scale domain and consumer layout for native \mxfp{}, global \nvfp{}, and cooperative-thread-array-local \nvfp{}. We evaluate Llama-3-family 8B pretraining through 160 billion tokens using bfloat16 output projections and compiled cross entropy. In matched same-accelerator probes, bfloat16 and Transformer Engine \nvfp{} reach 18.8K and 27.6K tokens/s/GPU, while our fastest custom route reaches 37.9K. \mxfp{} with row-gradient stochastic rounding and fixed-sign 32-value Hadamard weight-gradient preconditioning reaches 37.2K tokens/s/GPU (86.3% bfloat16 model FLOP utilization) and ends 2.11% above the raw bfloat16 training-loss endpoint. A Transformer Engine recipe with four final bfloat16 blocks ends 0.87% above bfloat16 at 27.1K tokens/s/GPU. Downstream rankings differ from training-loss rankings, showing that FP4 outcomes depend jointly on scale contract, operand, and execution path.

Robert Hu

GRAPHCORE RESEARCH

Low-Precision Training LLM Quantization Mixed-Precision Quantization Language Model Pretraining

Ask PDF

Abstract

Efficient Transformer Inference

Explore similar work

Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0

FPTQuant: Function-Preserving Transforms for LLM Quantization

Format-Aware Fusion for Fast FP4 Pretraining