cs.AINov 6, 2026

Fused-Softmax for efficient quantized attention

Authors: Tianxing Wu, Juuso Korhonen, Jesus Carabano Bravo

Organizations: Advanced Micro Devices, Inc., Austin, TX, USA

Abstract

Low-precision matrix throughput on modern GPUs is growing far faster than vector-arithmetic throughput. Attention therefore cannot realize the available GEMM performance because softmax has become the pipeline bottleneck. We introduce Fused-Softmax Attention, a training-free algorithm that reformulates and approximates softmax to fold its dominant operations into the attention GEMMs, moving work from vector units to matrix cores. Implemented for the AMD CDNA 4 architecture, our MXFP8, MXFP6 and MXFP4 kernels reach approximately 3.0, 4.0 and 4.6 PFLOP/s on an AMD Instinct MI355X GPU, yielding speedups of 2.1×2.1\times, 2.8×2.8\times and 3.2×3.2\times over the BF16 attention. Importantly, the additional error introduced by Fused-Softmax remains below the error floor set by quantizing the GEMM operands. On Wan2.2-T2V-A14B, a hybrid MXFP6/4 schedule delivers up to 2.05×2.05\times end-to-end speedup while preserving the visual quality of the outputs. The kernels ship in AMD's open-source AITER library, and they powered AMD's MLPerf Inference v6.1 Wan2.2 submission, which ran 1.7×1.7\times faster than AMD's previous-round result on the same hardware. Our results show that redesigning softmax to utilize the matrix pipeline enables attention to capitalize on continued growth in low-precision matrix throughput.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ZYFA: Speed and Accuracy of FP8 FlashAttention on AMD CDNA4

    Nov 5, 2026Quentin Anthony, Ganesh Bikshandi, Praneeth Medepalli +1Efficient AttentionGPU Acceleration

  2. EFQ-Softmax: Exp-Free Quantization for Softmax

    Sep 9, 2026Haohui Han, Yuming Wan, Hongni Wang +4Efficient Transformer InferenceTransformer Attention