Attention is a core building block of frontier models and often dominates compute-bound prefill at long context. FP8 matrix instructions offer twice the peak throughput of BF16, but they accelerate only matrix multiplication, and the softmax between the two matrix multiplications of attention runs on vector units that FP8 does not accelerate. On the AMD Instinct MI355X, the vendor's AITER kernels reach 39% of peak FP8 matrix throughput at 32K tokens, compared with 49% in BF16. Attention is also more numerically sensitive to quantization than feed-forward layers, and unlike NVIDIA's Hopper and Blackwell GPUs, the MI355X does not track the completion of matrix instructions in hardware, so overlapping the softmax with the matrix multiplications depends on instruction order. We present ZYFA, prefill kernels for the MI355X built on three techniques: (1) computing each FP8 probability directly from its score with a multiply-add and an integer conversion instead of an exponential; (2) a ping-pong schedule in which one group of threads computes its softmax while another executes its matrix multiplications; and (3) scaling each block of probabilities up by a power of two before rounding, and normalizing by the sum of the rounded probabilities. On attention inputs captured from three models and timed at sustained clock speeds, ZYFA achieves 1.16-1.76x the throughput of AITER. On the Gaussian inputs of kernel benchmarks at 32K tokens, it achieves 1.26-1.37x at head dimension 128 and 1.99x for DeepSeek-style multi-head latent attention. The sustained clock of the MI355X depends on operand values, so throughput on random inputs does not reliably predict throughput on real model activations. At head dimension 128, ZYFA's numerical error is within a few percent of that of Composable Kernel and Triton and 62-78% lower than that of AITER's. The source code is available at https://github.com/Zyphra/zyfa.
Figures & tables
Property
Value per GPU
Compute units
256 in 8 XCDs, 4 SIMDs each
Wave, VGPRs
64 lanes, 512 VGPRs per lane per SIMD
LDS
160 KB per CU
L2 cache
4 MB per XCD
Infinity Cache
256 MB, shared by the XCDs
HBM3E
288 GB, 8 TB/s
TABLE I : Properties of one MI355X from ( Advanced Micro Devices, 2025b ; Advanced Micro Devices, 2025a ) . The peaks assume the maximum clock. VGPRs are the vector registers of a wave, XCDs are the eight chiplets, and LDS is the shared memory of a compute unit (CU). Our system caps the board power at the TBP.
Term
Meaning
CU, SIMD
A compute unit (CU) has four SIMD units, each of which issues the instructions of its waves
wave
64 threads that issue instructions together, AMD’s counterpart of a 32 -thread NVIDIA warp
MFMA
Matrix fused multiply-add, the matrix instruction that runs on the matrix pipe, the SIMD’s matrix unit
VALU
The vector pipe, the SIMD’s vector unit, which runs elementwise arithmetic such as the softmax
LDS
Local data share, the 160 KB shared memory of a CU
XCD
One of the eight chiplets of the GPU, each with its own L2 cache
TABLE II : Terms used throughout the paper.
work for one wave and one block
cycles
matrix pipe
QK⊤ and PV in FP8
1024
QK⊤ and PV in BF16
2048
row sum of the FP8 probabilities
64
vector pipe
row maximum ( 32 three-input maxima)
128
TABLE III : In FP8 the softmax needs about as many vector-pipe cycles as the matrix products need matrix-pipe cycles, so the two must overlap almost perfectly to keep the matrix pipe busy, while in BF16 the matrix products take more than twice as long as the softmax. The table gives pipe cycles for one wave (a group of 64 threads) that computes 32 queries against one 128 -key block at dqk=dv=128 , and each of its 64 lanes holds 64 of the 4096 scores. Matrix cycles, on the pipe that runs matrix (MFMA) instructions, follow from the matrix rate per cycle, and vector cycles, on the pipe that runs elementwise instructions, follow from the measured costs in Section III-A . Exact P, the standard method, evaluates the exponential and converts the result to e4m3 or BF16, and the packed BF16 conversion occupies the vector pipe for 4 cycles instead of 8 . Lean P, ZYFA’s method, writes the e4m3 bit pattern of each probability with one multiply-add and one integer conversion (Section IV-B1 ). Rescaling the output adds vector work when the row maximum grows.
cycles per SIMD
base
AITER
matrix pipe (packed FP32)
1288 (264)
1156 (132)
vector pipe
1210
1303
instruction issue
1027
891
LDS
772
516
serialized (matrix + vector)
2498
2459
achieved
2470
2051
TABLE IV : Both kernels keep the vector pipe about as busy as the matrix pipe, and neither overlaps the two fully. The table gives pipe cycles per SIMD at S=32 K for FP8 multi-head attention at dqk=128 , for two units of work of one wave on 32 queries and 64 keys each, which together equal the work of one wave on one 128 -key block in Table III . AITER is AITER’s FP8 assembly kernel. Base is an earlier HIP version of ZYFA’s FP8 kernel with a Schraudolph exponential (an exponential that writes the bits of an FP32 number directly from the score) and an FP32 row sum, and its instruction mix comes from S=8 K traces. The matrix row includes the cycles in which packed FP32 instructions block the matrix pipe, given in parentheses. The serialized row is the sum of matrix and vector cycles, which the work would take if the two never overlapped, and the achieved row gives the cycles that each kernel actually takes. SCLK is the shader clock.
Fig. 2 : Thread traces showing that AITER’s FP8 kernel leaves the matrix pipe idle while its waves compute exponentials and wait at barriers, and that ZYFA’s FP8 kernel keeps the matrix pipe busy most of the time. A thread trace records every instruction that a wave issues, cycle by cycle, here with the thread-trace mode of AMD’s rocprofv3 profiler. Each panel shows the two waves that share one SIMD over 2500 cycles in the middle of the loop over key blocks, where every block repeats the same work, in the FP8 kernels at dqk=128 ( S=8 K tokens, H=32 heads, multi-head attention). Each wave row shows the class of each instruction that the wave issues, and the matrix-pipe row below it shows when that wave’s MFMAs run. A vector instruction runs on the vector pipe during its bar in the wave row, so the wave rows already show when the vector pipe works. An MFMA keeps running on the matrix pipe for 32 to 64 cycles after its short bar in the wave row, while the wave issues other instructions, so the matrix-pipe rows show that work separately.
workload
GHz
kW
PF/s
BF16 MFMA 32 × 32
1.71
1.25
1.79
BF16 MFMA 16 × 16
1.92
1.27
1.99
FP8 MFMA 32 × 32
2.07
1.29
4.34
FP8 MFMA 16 × 16
2.20
1.31
4.60
FP4 MFMA 16 × 16
1.91
1.26
7.86
VALU only
2.39
0.66–0.70
–
TABLE V : Under sustained load the MFMA loops run below the 2.4 GHz clock that the datasheet peaks assume, so each reaches less than its datasheet throughput. The table gives empirical peaks, the sustained throughput of microbenchmark loops on all 256 compute units (CUs) with two waves per SIMD and operands that change every instruction, over about 2.5 s of back-to-back launches. The columns give the shader clock (SCLK) in GHz, the board power in kW, and the throughput in PFLOP/s. The synthetic attention loop has the per-tile counts of MFMAs, reads from the on-chip shared memory (LDS), loads into it (DMA), and barriers of ZYFA’s FP8 kernel at dqk=128 . The BF16 rows use v_mfma_f32_32x32x16_bf16 and v_mfma_f32_16x16x32_bf16 , the FP8 rows v_mfma_f32_32x32x64_f8f6f4 and v_mfma_f32_16x16x128_f8f6f4 with e4m3 operands, and the FP4 row v_mfma_f32_16x16x128_f8f6f4 with FP4 operands, all in their unscaled forms. The datasheet peaks assume 2.4 GHz.
Attention
Causal forward pass, without sliding-window, document, or block masks
Heads
MHA and GQA ( Hq a multiple of Hkv )
Layouts
BHSD and BSHD (strided)
Batch
Any B with equal sequence lengths, without packed variable-length batches
Lengths
M=N , a multiple of 256 (no KV-cache prefix or decoding)
Inputs
BF16, or e4m3 with one FP32 scale per tensor
Outputs
BF16 O and FP32 LSE
TABLE VI : Configurations that the ZYFA kernels support. MHA is multi-head attention and GQA grouped-query attention, in which Hq/Hkv query heads share each key-value head. BHSD and BSHD give the order of the batch, head, sequence, and head-dimension axes in memory, M and N are the numbers of queries and keys, and LSE is the log-sum-exp of each row of scores.
Fig. 7 : Thread traces showing that in BF16 the matrix products set the time of all four kernels of ZYFA and AITER, because they take more than twice as many cycles as the softmax (Table III ). A thread trace records every instruction that a wave issues, cycle by cycle. Each panel shows the two waves that share one SIMD over 4000 cycles in the middle of the loop over key blocks, where every block repeats the same work ( S=8 K tokens, H=32 heads, multi-head attention). Each wave row shows the class of each instruction that the wave issues, and the matrix-pipe row below it shows when that wave’s MFMAs run. A vector instruction runs on the vector pipe during its bar in the wave row, whereas an MFMA keeps running on the matrix pipe after its short bar, while the wave issues other instructions.
kernel
product
instruction
per kernel
FLOP/clock/SIMD
ZYFA, FP8 d=128
row sum of P
v_mfma_f32_16x16x128_f8f6f4
20
2041
QK⊤
v_mfma_f32_32x32x64_f8f6f4
64
2048
PV
v_mfma_scale_f32_32x32x64_f8f6f4
80
2048
ZYFA, FP8 dqk=192
row sum of P
v_mfma_f32_16x16x128_f8f6f4
10
2041
QK⊤
v_mfma_f32_32x32x64_f8f6f4
48
2048
PV
v_mfma_scale_f32_32x32x64_f8f6f4
40
2048
TABLE VII : FP8 matrix (MFMA) instructions of each FP8 kernel, read from the disassembled kernel binaries. The count is the number of times each instruction appears in the kernel binary, and the rate is in FLOP per clock per SIMD unit, measured with two waves (groups of 64 threads) per SIMD on all 256 compute units (CUs). ZYFA’s PV instruction applies the per-block scale of Section IV-B2 , a power of two for each block of 64 probabilities. Composable Kernel (CK) is AITER’s FP8 kernel at dqk=192 .
multi-head
grouped-query
kernel
2K
16K
32K
2K
16K
32K
spread
BF16 128
1.05
1.03
1.02
1.07
1.02
1.02
1.7%
BF16 192
1.04
1.01
1.00
1.07
1.00
1.00
1.7%
FP8 128
1.44
1.34
1.37
1.42
1.28
1.26
1.4%
FP8 192
2.02
1.89
1.99
1.94
1.82
1.88
0.8%
TABLE VIII : Throughput of ZYFA divided by that of AITER’s default kernel at batch size B=1 , so values above 1 mean that ZYFA takes less time. Inputs are Gaussian (random q , k , and v with standard deviation 0.1 ), and times are kernel-only (the device time of the kernel alone), averaged over repetitions. Rows give the precision and the query-key head dimension dqk . At dqk=192 the FP8 baseline is Composable Kernel (CK). The last column is the spread of ZYFA’s multi-head throughput over repetitions at 32 K. Figure 8 gives all sequence lengths, and Table IX gives B=4 .
Fig. 8 : Throughput of ZYFA and AITER against sequence length on Gaussian inputs (random q , k , and v with standard deviation 0.1 ). Times are kernel-only, the device time of the kernel alone without the time that the host takes to launch it. AITER is AITER’s default kernel for each configuration, which at dqk=192 is its OPUS kernel in BF16 and Composable Kernel in FP8. The BF16 panels also show AITER’s Triton kernel and, at dqk=192 , AITER’s assembly kernel.
S=8 K
S=32 K
kernel
B=1
B=4
B=1
B=4
BF16 dqk=128
1.03
1.00
1.02
1.03
FP8 dqk=128
1.35
1.41
1.37
1.38
BF16 dqk=192
1.00
1.03
1.00
1.00
FP8 dqk=192
1.88
1.94
1.99
1.99
TABLE IX : Throughput of ZYFA divided by that of AITER’s default kernel at batch sizes B=1 and B=4 , for multi-head attention on Gaussian inputs, kernel-only. Values above 1 mean that ZYFA takes less time.
time ratio
SCLK (MHz), capt./Gauss.
model
capt.
Gauss.
ZYFA
AITER
Mistral
FP8
1.16
1.30
1628/1686
2108/1991
Qwen3
FP8
1.17
1.29
1629/1687
2048/1993
DeepSeek
FP8
1.76
1.84
1742/1803
2219/2250
Mistral
BF16
1.02
1.03
1590/1644
1446/1502
Qwen3
BF16
1.01
1.02
1586/1645
1438/1503
TABLE X : ZYFA and AITER on one layer per model at S=16 K (Mistral-7B layer 20, Qwen3-8B layer 18, DeepSeek-V2-Lite layer 13). The captured columns use the q , k , and v that the model produces on WikiText-2 text, and the Gaussian columns use random inputs of the same shape. Each run launches one kernel back to back for seconds and is timed only after the GPU clock has settled (sustained clocks). The time ratio is AITER’s time divided by ZYFA’s time, so values above 1 mean that ZYFA takes less time. SCLK is the shader clock of the slowest of the eight chiplets, which sets the kernel time.
FP8
BF16
model
AITER
ZYFA
CK
Triton
ZYFA
Mistral-7B
6.41×10−2
0.22
0.22
0.21
1.02
Qwen3-8B
3.17×10−2
0.38
0.37
0.37
1.01
DeepSeek-V2-Lite
1.02×10−2
0.82
1.00
–
1.00
TABLE XI : Output error of one attention call, measured against attention computed in fp64 on the same quantized inputs, so it counts only the error that each kernel adds. Values are means over all layers of each model on WikiText-2 at S=4 K. The AITER column gives the relative RMS error of AITER’s default FP8 kernel, which is Composable Kernel (CK) at dqk=192 . The other FP8 columns give each kernel’s error divided by AITER’s, so values below 1 mean a smaller error. AITER’s Triton kernel does not support FP8 when dqk=dv . The BF16 column gives ZYFA’s BF16 error divided by the error of AITER’s BF16 kernel.
layer
between launches
AITER
ZYFA
Mistral-7B L20, 4K
nothing
0/200
0/200
Mistral-7B L20, 4K
BF16 GEMM
5/2000
0/2000
Mistral-7B L20, 4K
BF16 attention
0/200
0/200
Qwen3-8B L6, 4K
nothing
0/200
0/200
Qwen3-8B L6, 4K
BF16 GEMM
3/2000
0/2000
Qwen3-8B L6, 4K
BF16 attention
0/200
0/200
TABLE XII : Number of launches, out of the total after the slash, whose output differs bitwise from the first launch on the same captured FP8 inputs. Between launches the GPU runs nothing, a BF16 GEMM (matrix multiplication), or a BF16 attention kernel. AITER is AITER’s FP8 assembly kernel.
error of one call
time / all-FP8
transform
transform
input
total
KL
dNLL [95% interval]
fused
separate
ms at 16K
Mistral-7B
none
8.27e-02
8.32e-02
6.05e-03
+5.4e-03 [+5.0e-03, +5.8e-03]
1.000
1.000
–
rotation
4.41e-02
4.52e-02
2.90e-03
+2.6e-03 [+2.4e-03, +2.9e-03]
1.001
1.089
2.51
rotation, K smoothing
4.55e-02
4.65e-02
2.62e-03
+2.2e-03 [+2.0e-03, +2.5e-03]
1.006
1.165
3.97
Qwen3-8B
TABLE XIII : Effect of rotating Q and K with a Hadamard matrix (rotation) and of subtracting the mean key of each head (K smoothing) before FP8 quantization, for all-FP8 ZYFA attention. The error of one call is the relative RMS error of the attention output O against fp64 attention, from quantizing the inputs alone (input) and in total. KL is the per-token KL divergence from the model with BF16 attention, and dNLL the change in negative log-likelihood per token, both on the 48 PG-19 evaluation documents. Times are relative to all-FP8 attention without transforms. The fused column includes one quantization pass that also applies the transforms. The separate column runs the transforms as PyTorch operations and includes no quantization. The last column is the time of the separate transforms per forward pass at 16 K.
Mistral-7B
Qwen3-8B
DeepSeek-V2-Lite
score
unit
5%
10%
20%
5%
10%
20%
5%
10%
20%
local-error oracle
heads
0.94
0.91
0.83
0.94
0.91
0.83
0.98
0.94
0.87
local-error oracle
row blocks
0.93
0.89
0.82
0.93
0.90
0.81
0.95
0.91
0.84
ge2
heads
0.52
0.43
0.31
0.66
0.57
0.43
0.41
0.34
0.25
layer KL
layers
0.67
0.61
0.45
0.74
0.66
0.54
0.85
0.64
0.33
logit spread
heads
0.95
0.92
0.98
0.87
0.79
0.66
0.86
0.70
0.58
TABLE XIV : How well each score chooses the units (heads, 128 -row blocks, or layers) that run in BF16. Each value is the KL divergence from the BF16 model that remains when the top 5 , 10 , or 20% of units by the score run in BF16, divided by the KL divergence of all-FP8 attention (lower is better), on 12 evaluation documents per model with the exact-P build of ZYFA’s FP8 kernel. The local-error oracle ranks units by the true FP8 error of their own attention output. ge2 ranks heads by their gradient-weighted error g2e2 , and layer KL ranks layers by how much FP8 attention in that layer alone changes the output of the model. Both use calibration documents (Section VI-C ), and the scores below the rule are computed at inference time. A row block has 128 query rows, and its BF16 share of the causal work differs slightly from the nominal share. On the 48 documents of Figure 24 with lean P, ge2 and layer KL give values within 0.02 of these.
share of
attention weight on
mean
top-channel share
rank
layer.head
ge2
local error
key 0
last 64
distance
entropy
Q
K
type
1
4.9
6.6%
0.01%
0.49
0.25
3108
1.87
0.12
0.18
long-range
2
4.8
6.2%
0.01%
0.51
0.24
3159
1.87
0.14
0.18
sink
3
4.11
2.3%
0.01%
0.55
0.20
3322
1.65
0.12
0.18
sink
4
5.11
2.0%
0.02%
0.23
0.06
2547
4.85
0.14
0.15
long-range
5
2.21
1.6%
0.00%
0.49
0.03
3138
2.41
0.11
0.09
long-range
TABLE XV : Heads of Mistral-7B with the largest ge2 score, the gradient-weighted FP8 error g2e2 of a head (Section VI-C ), on the calibration documents. The share columns give the head’s share of the ge2 score and of the local FP8 error. The attention-weight columns give the mean weight on key 0 and on the last 64 keys over the query rows from 256 on, on two calibration documents. Qwen3-8B’s documents have no BOS token, so its key 0 is the first text token. The mean distance between query and key is weighted by attention and given in tokens, and the entropy of the attention row is given in nats. The top-channel share is the share of the energy of post-RoPE Q and K in their largest channel, pooled over the tokens of a document and averaged over eight calibration documents. A head is a sink head with weight on key 0 of at least 0.5 , local with weight on the last 64 keys of at least 0.5 , long-range with neither and a mean distance of at least 1024 , and an outlier head with a top-channel share of at least 0.2 .
share of
attention weight on
mean
top-channel share
rank
layer.head
ge2
local error
key 0
last 64
distance
entropy
Q
K
type
1
0.6
3.3%
0.00%
0.00
0.88
122
3.26
0.08
0.97
local, outlier
2
0.11
2.5%
0.00%
0.00
0.87
138
3.14
0.07
0.97
local, outlier
3
7.7
2.0%
0.01%
0.69
0.02
3396
1.17
0.28
0.27
sink, outlier
4
10.2
1.9%
0.07%
0.36
0.56
1570
1.93
0.31
0.66
local, outlier
5
7.5
1.7%
0.01%
0.76
0.08
3281
0.92
0.24
0.27
sink, outlier
TABLE XVI : Heads of Qwen3-8B with the largest ge2 score (gradient-weighted FP8 error), with the columns of Table XV .
share of
attention weight on
mean
top-channel share
rank
layer.head
ge2
local error
key 0
last 64
distance
entropy
Q
K
type
1
0.9
5.4%
0.04%
0.00
0.95
114
0.89
0.34
0.55
local, outlier
2
2.9
5.1%
0.05%
0.00
0.96
93
2.96
0.21
0.65
local, outlier
3
1.8
5.0%
0.01%
0.00
0.94
65
3.23
0.24
0.67
local, outlier
4
4.7
3.6%
0.14%
0.41
0.56
1746
2.11
0.35
0.57
local, outlier
5
5.2
2.7%
0.61%
0.17
0.74
1073
1.16
0.30
0.68
local, outlier
TABLE XVII : Heads of DeepSeek-V2-Lite with the largest ge2 score (gradient-weighted FP8 error), with the columns of Table XV .
all policies with mixed layers
10% of heads in BF16 by ge2
model
S
policies
copy ms
% of attention
mixed layers
copy ms
attention ms
%
Mistral-7B
8K
28
0.16–5.38
1.2–35.2
23
3.49
11.36
30.8
Mistral-7B
16K
28
0.31–10.77
0.7–22.3
23
7.15
35.68
20.0
Mistral-7B
32K
28
0.60–21.18
0.3–12.1
23
13.85
129.21
10.7
Qwen3-8B
8K
28
0.15–5.39
1.0–31.9
20
3.04
12.40
24.5
Qwen3-8B
16K
28
0.31–11.01
0.5–20.5
20
6.24
39.90
15.6
TABLE XVIII : Time per forward pass of the copies that gather the heads of each precision into contiguous tensors and scatter their outputs back, in mixed-precision layers (layers with heads in both BF16 and FP8). The left group covers every policy with at least one mixed layer and gives the range over policies. The right group is the policy with the 10% of heads with the largest ge2 score in BF16. The attention times of Section VI-C exclude the copies.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
model
8 K
16 K
32 K
Mistral-7B
FP8
1.27 (1.22–1.35)
1.26 (1.25–1.29)
1.24 (1.23–1.27)
Qwen3-8B
FP8
1.26 (1.22–1.32)
1.25 (1.21–1.27)
1.22 (1.19–1.23)
DeepSeek
FP8
1.88 (1.83–1.91)
1.75 (1.72–1.76)
1.77 (1.75–1.81)
Mistral-7B
BF16
1.03 (1.01–1.04)
1.03 (1.02–1.03)
1.03 (1.02–1.03)
Qwen3-8B
BF16
1.01 (0.96–1.03)
1.02 (0.98–1.03)
1.02 (0.98–1.03)
DeepSeek
BF16
1.00 (0.99–1.01)
1.00 (0.99–1.01)
1.00 (0.99–1.01)
Appendix
TABLE XIX : AITER’s time divided by ZYFA’s time for the attention of every layer, measured inside a full forward pass of the model at each sequence length, as the median and, in parentheses, the range over the layers of each model, on two PG-19 documents. Values above 1 mean that ZYFA takes less time. DeepSeek is DeepSeek-V2-Lite.
Low-precision matrix throughput on modern GPUs is growing far faster than vector-arithmetic throughput. Attention therefore cannot realize the available GEMM performance because softmax has become the pipeline bottleneck. We introduce Fused-Softmax Attention, a training-free algorithm that reformulates and approximates softmax to fold its dominant operations into the attention GEMMs, moving work from vector units to matrix cores. Implemented for the AMD CDNA 4 architecture, our MXFP8, MXFP6 and MXFP4 kernels reach approximately 3.0, 4.0 and 4.6 PFLOP/s on an AMD Instinct MI355X GPU, yielding speedups of 2.1×, 2.8× and 3.2× over the BF16 attention. Importantly, the additional error introduced by Fused-Softmax remains below the error floor set by quantizing the GEMM operands. On Wan2.2-T2V-A14B, a hybrid MXFP6/4 schedule delivers up to 2.05× end-to-end speedup while preserving the visual quality of the outputs. The kernels ship in AMD's open-source AITER library, and they powered AMD's MLPerf Inference v6.1 Wan2.2 submission, which ran 1.7× faster than AMD's previous-round result on the same hardware. Our results show that redesigning softmax to utilize the matrix pipeline enables attention to capitalize on continued growth in low-precision matrix throughput.
Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies dominate once its matrix products shrink. We address this with Direct-P for noncausal inference and a causal path that passes the forward quantization directly into backward. Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13× the bfloat16 (BF16) forward throughput on an NVIDIA GB200. The causal path reconstructs probabilities from saved quantized queries and keys and uses 8-bit floating-point (FP8) gradient operands, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14×. Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.
Attention remains a substantial computational cost in Transformer training, particularly at long context lengths. NVIDIA Blackwell GPUs provide high-throughput low-precision matrix multiplication, but exploiting it in attention requires addressing quantization overhead, causal information leakage through shared scales, and numerical inconsistencies between the forward and backward passes. We present FLAMIA, a family of attention kernels for Blackwell with MXFP8 and mixed NVFP4/FP8 training paths. Our design combines token-block scaling that reuses quantized operands across matrix orientations, targeted safeguards against scale-induced leakage, and consistent forward and backward computations. Fused quantization and architecture-specific kernel optimizations reduce overhead, while document-local scaling and workload-aware scheduling support variable-length documents directly in packed layouts without repacking. Across various causal packed-document workloads, the MXFP8 path accelerates the combined forward and backward attention call, including quantization, by factors of 1.10--1.40× on B200 and 1.06--1.42× on B300 relative to the fastest evaluated baseline for each workload. We validate training stability and quality preservation on pretraining Llama-style models. These results show how jointly designing quantization, training numerics, and document scheduling translates low-bit hardware throughput into practical attention acceleration.