Low-precision matrix throughput on modern GPUs is growing far faster than vector-arithmetic throughput. Attention therefore cannot realize the available GEMM performance because softmax has become the pipeline bottleneck. We introduce Fused-Softmax Attention, a training-free algorithm that reformulates and approximates softmax to fold its dominant operations into the attention GEMMs, moving work from vector units to matrix cores. Implemented for the AMD CDNA 4 architecture, our MXFP8, MXFP6 and MXFP4 kernels reach approximately 3.0, 4.0 and 4.6 PFLOP/s on an AMD Instinct MI355X GPU, yielding speedups of 2.1×, 2.8× and 3.2× over the BF16 attention. Importantly, the additional error introduced by Fused-Softmax remains below the error floor set by quantizing the GEMM operands. On Wan2.2-T2V-A14B, a hybrid MXFP6/4 schedule delivers up to 2.05× end-to-end speedup while preserving the visual quality of the outputs. The kernels ship in AMD's open-source AITER library, and they powered AMD's MLPerf Inference v6.1 Wan2.2 submission, which ran 1.7× faster than AMD's previous-round result on the same hardware. Our results show that redesigning softmax to utilize the matrix pipeline enables attention to capitalize on continued growth in low-precision matrix throughput.
Figures & tables
Unit
CDNA 3
CDNA 4
Gain
Matrix, BF16
2048
4096
2×
Matrix, FP8
4096
8192
2×
Matrix, MXFP6/FP4
—
16384
—
Vector ALU
128
128
1×
Exponential unit
16
32
2×
LDS read (bytes)
128
256
2×
Table 1: Throughput evolution on the last two CDNA generations. Matrix, vector arithmetic logic unit (VALU), and exponential rows are ops/clock/CU; the local data share (LDS) row is bytes/clock/CU.
Figure 1: Eight-wave ping-pong schedule on four SIMDs. Waves 0 – 3 and 4 – 7 form phase-shifted groups: each wave overlaps its matrix operations with its SIMD partner’s softmax, and barriers preserve the phase offset across tiles.
FP8
FP6
FP4
THBM
2517
1887
1258
TLDS
1024
768
512
TGEMM
2048
1024
1024
TVALU
2304
2304
2304
TVALU/TGEMM
1.125×
2.25×
2.25×
Table 2: Per-tile roofline, in clock cycles.
Figure 2: One SIMD pair in the Fused-Softmax schedule. While one wave issues the denominator, PV and the next QK matrix operations, its partner evaluates the row maximum, rollback gate and two-stage conversion. Prefetch and drain operations use alternating LDS slots; the rollback path below the timeline runs very rarely, only when the gate fires.
components
BF16
MXFP8
MXFP6
MXFP4
ℓ as GEMM
frozen max
Schraudolph
proxy
err
cos
err
cos
err
cos
err
cos
–
–
–
5.39
0.99855
6.17
0.99810
20.45
0.97890
✓
–
0.026
1.00000
5.39
0.99855
6.17
0.99810
20.45
0.97890
✓
✓
–
0.310
1.00000
5.42
0.99853
6.17
0.99810
20.45
0.97890
✓
✓
✓
U16
1.798
0.99984
5.69
0.99838
6.46
0.99791
20.53
0.97872
✓
✓
✓
U32
1.798
0.99984
5.69
0.99838
6.46
0.99791
20.53
0.97872
Table 4: Component ablation on normal inputs ( b=1 , h=24 , s=32768 , d=128 , ten seeds): global relative error in % and cosine similarity against BF16 attention with the conventional softmax. Each row enables the marked components. The bolded row is our complete Fused-Softmax pipeline; the last two rows vary the Schraudolph proxy of the complete pipeline.
ours
related work
tier
kernel
TFLOP/s
cosine
method
TFLOP/s
cosine
FP8
mxfp8
2978
0.99838
Attn-QAT MXFP8/FP8
2362
0.9985
∼ FP6
mxfp6
3973
0.99791
Attn-QAT NVFP4/FP8
2677
0.9899
FP4
mxfp4
4586
0.97872
Direct-P
2945
0.9429
Table 5: Throughput and BF16-reference cosine on normal input ( b=1 , h=24 , s=32768 , d=128 ): our kernels versus published Attn-QAT Zhang et al. (2026b) and Direct-P Hu (2026) results.
Figure 3: Forward throughput against sequence length at b=1 , h=24 , d=128 , non-causal: our kernels on MI355X next to the BF16 FlashAttention-4 and Attn-QAT throughput published on GB300 Zhang et al. (2026b) .
latency (s)
VBench
PSNR ↑
SSIM ↑
LPIPS ↓
BF16 GEMMs
FP8/4 GEMMs
BF16
71.83
reference
197.5
173.3
MXFP8
71.78
17.37
0.584
0.324
129.4 ( 1.53× )
105.9 ( 1.64× )
MXFP6
71.85
16.32
0.546
0.363
115.5 ( 1.71× )
88.9 ( 1.95× )
MXFP4
71.13
14.55
0.477
0.486
106.0 ( 1.86× )
82.6 ( 2.10× )
hybrid
72.01
16.31
0.540
0.382
108.0 ( 1.83× )
84.4 ( 2.05× )
Table 6: Wan2.2-T2V-A14B: VBench score and PSNR (dB), SSIM and LPIPS against the BF16 videos over the 248 prompts. The last two columns report end-to-end latency (s) of the motorcycle prompt under the BF16 and optimized model configurations of Section 5.3.3 ; parentheses give the speedup over BF16 attention in the same configuration.
Figure 4: Wan2.2-T2V-A14B, frame 40 of the cat (top) and motorcycle (bottom) prompts with each self-attention kernel.
Figure 5: Motorcycle prompt across two additional models and all self-attention kernels.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Wan2.2-T2V-A14B, motorcycle prompt, at the MLPerf text-to-video settings with the scheduler’s default flow shift of 3 (left) and with 12 (right), frames 0 , 40 and 80 , for BF16 and MXFP6 attention. Only the flow shift differs between the two sides.
Wan2.2
Subject consistency
Background consistency
Motion smoothness
Dynamic degree
Appearance style
Scene
Overall
PSNR
SSIM
LPIPS
BF16
92.64
93.64
96.77
93.06
20.74
34.16
71.83
reference
MXFP8
93.22
93.71
96.82
93.06
20.77
33.14
71.78
17.37
0.584
0.324
MXFP6
93.03
93.88
96.72
91.67
20.58
35.25
71.85
16.32
0.546
0.363
MXFP4
92.66
93.82
97.38
90.28
20.68
31.98
71.13
14.55
0.477
0.486
hybrid
93.09
94.01
97.06
93.06
20.42
34.45
72.01
16.31
0.540
0.382
Appendix
Table 7: VBench scores and closeness to the BF16 videos for Wan2.2-T2V-A14B, flow shift 12 . Each VBench dimension is the mean over the prompts it applies to, on the 0 – 100 scale; Overall is the mean of the six dimensions, the benchmark’s accuracy score. PSNR, SSIM and LPIPS compare every fourth frame of each video with the same frame of the BF16 video of the same prompt, averaged over the 248 prompts.
reach
relative error over one doubling interval of f
added
proxy
M
E
of P
min
max
RMS
by f^
continuous
∞
—
—
−3.939%
1.966%
1.770%
—
U32 / FP32
23
8
2128
−3.939%
1.966%
1.770%
0.000%
U16 / FP16
10
5
216
−3.972%
2.000%
1.770%
0.020%
U16 / BF16
7
8
2128
−4.199%
2.242%
1.777%
0.162%
U8 / E4M3
3
4
28
−8.012%
6.398%
3.141%
2.595%
Appendix
Table 8: What each proxy width buys. M and E are the mantissa and exponent widths of the pattern the proxy carries. The exponent-scale reach of P is approximately 22E−1 and depends on E alone. The error columns depend on M alone and are swept over one doubling interval of f ; the last column is the part of the RMS contributed by f^ , separated in quadrature from the 1.770% the ramp carries regardless.
Figure 7: The ramp error, independent of any datatype. Top: over one doubling interval the mantissa field interpolates 2x linearly, the chord from (0,1) to (1,2) . Bottom: the resulting relative error is periodic with period one doubling.
e
pattern
Δ
read as FP16
reduction
gate
0 – 247
0x0000 – 0x7BFF
<121
positive finite
correct
fires iff above the cap
248 – 255
0x7C00 – 0x7FFF
121 – 128
+∞ or + NaN
kept ( ∞ ) or propagated (NaN)
fires
256 – 504
0x8000 – 0xFC00
129 – 377
negative finite, and −∞
dropped
misses
504 – 511
0xFC01 – 0xFFFF
377 – 384
− NaN
propagates NaN
fires
Appendix
Table 9: How v_pk_maximum3_f16 reads a Stage-A pattern. Here e=pattern≫7 is the pattern’s nine-bit sign-plus-exponent code, and Δ=e−127 its offset from the retained reference in doublings. The gate is an unsigned compare against the cap, Δ=32 , of Section 4.1 . Only the third band misreads; its upper boundary falls inside e=504 , so the bands are delimited by pattern.
Attention is a core building block of frontier models and often dominates compute-bound prefill at long context. FP8 matrix instructions offer twice the peak throughput of BF16, but they accelerate only matrix multiplication, and the softmax between the two matrix multiplications of attention runs on vector units that FP8 does not accelerate. On the AMD Instinct MI355X, the vendor's AITER kernels reach 39% of peak FP8 matrix throughput at 32K tokens, compared with 49% in BF16. Attention is also more numerically sensitive to quantization than feed-forward layers, and unlike NVIDIA's Hopper and Blackwell GPUs, the MI355X does not track the completion of matrix instructions in hardware, so overlapping the softmax with the matrix multiplications depends on instruction order. We present ZYFA, prefill kernels for the MI355X built on three techniques: (1) computing each FP8 probability directly from its score with a multiply-add and an integer conversion instead of an exponential; (2) a ping-pong schedule in which one group of threads computes its softmax while another executes its matrix multiplications; and (3) scaling each block of probabilities up by a power of two before rounding, and normalizing by the sum of the rounded probabilities. On attention inputs captured from three models and timed at sustained clock speeds, ZYFA achieves 1.16-1.76x the throughput of AITER. On the Gaussian inputs of kernel benchmarks at 32K tokens, it achieves 1.26-1.37x at head dimension 128 and 1.99x for DeepSeek-style multi-head latent attention. The sustained clock of the MI355X depends on operand values, so throughput on random inputs does not reliably predict throughput on real model activations. At head dimension 128, ZYFA's numerical error is within a few percent of that of Composable Kernel and Triton and 62-78% lower than that of AITER's. The source code is available at https://github.com/Zyphra/zyfa.
Low-bit attention accelerates Transformer inference by moving the QK⊤ and PV matrix multiplications to FP8 or FP4 matrix engines. However, the softmax path often evaluates shifted-score exponentials in higher precision, forms a temporary probability block, and quantizes it before low-bit PV multiplication. This exp-then-quantize path creates a mismatch between a high-precision probability producer and a low-bit matrix consumer. We propose EFQ-Softmax (Exp-Free Quantization for Softmax), a low-bit probability-generation method that directly maps shifted attention scores to block-scaled E2M1 operands. For each microscaling block, EFQ-Softmax selects an exponent-only scale from the local maximum, maps the shifted scores to a normalized residual domain, and generates nonnegative E2M1 probability codes using a single affine rule. The resulting operand is used consistently in both the PV numerator update and the P1 denominator update. The FlashAttention-style row-maximum update, historical rescaling, high-precision accumulation, and final normalization remain unchanged. We evaluate end-to-end quality on Qwen3-8B, Qwen3-VL-8B-Instruct, and WAN2.2-TI2V-5B, and separately measure kernel-level performance on the A5 vector unit. EFQ-Softmax improves the Qwen3-8B seven-task mean from 0.6749 with MXFP4 to 0.6773 and the Qwen3-VL nine-task mean from 0.7826 to 0.8000. On WAN2.2, it maintains temporal consistency and visual quality comparable to the FP16 and MXFP4 baselines under VBench. On the A5 vector unit, EFQ-Softmax reduces the vector-stage latency of the fused probability-generation kernel by 40.33% on average across sequence lengths from 16K to 128K. These results show that direct low-bit probability generation can replace the conventional exp-then-quantize path while preserving end-to-end model quality.
Haohui Han, Yuming Wan, Hongni Wang +4
Xi'an Jiaotong University · Huawei Technologies Co., Ltd · Shandong University of Finance and Economics
Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies dominate once its matrix products shrink. We address this with Direct-P for noncausal inference and a causal path that passes the forward quantization directly into backward. Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13× the bfloat16 (BF16) forward throughput on an NVIDIA GB200. The causal path reconstructs probabilities from saved quantized queries and keys and uses 8-bit floating-point (FP8) gradient operands, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14×. Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.