Authors: Jian-Jia Chen, Brian J. Chan, Wen-Hung Lee, Yi-Ting Li, Po-Shen Chen, Kuan-Te Kuo, Hsien-Cheng Huang, Cody Hao Yu, Chi-Chih Chang, Mohamed S. Abdelfattah, Kai-Chiang Wu
Organizations: National Yang Ming Chiao Tung University, Hsinchu, Taiwan · National Taiwan University, Taipei, Taiwan · OpenAI, San Francisco, CA, USA · Cornell University, USA
Blackwell tensor cores support native inference with 4-bit weights and activations (W4A4), but NVFP4 leaves a perplexity (PPL) gap to BF16. Choosing a format for each 16-element block can improve accuracy, but current matrix multiply instructions cannot execute choices this fine. Applying these rules at hardware-compatible tiles loses most of their benefit. We introduce FlipQuant, which selects E2M1 or E0M3 for each MMA weight tile. It keeps both quantized weight candidates fixed and learns a binary format map from end-to-end gradients of a bucketed KL loss to a BF16 teacher, re-evaluating them after each update. Native kernels execute 16 x 64 tiles on SM120 and 256 x 64 tiles on SM100 with median GEMM overheads of 1.9% and 1.7%, respectively. Across six language models and two corpora, FlipQuant at 16 x 64 closes 42.5% of NVFP4's PPL gap to BF16 on average, compared with 13.3% for FourOverSix. Code is available at https://github.com/brian030128/flipquant.
Figures & tables
Figure 1: Format selection at hardware granularity: the share of NVFP4’s PPL gap to BF16 closed (higher is better), averaged over WikiText-2, C4 and the six models of Table 2 . The per-block rules IF4 and MixFP4 need new hardware at 1×16 and fall below FourOverSix per MMA weight tile, the unit Blackwell tensor cores execute; FlipQuant selects formats per tile and runs natively (Appendix B ).
Method
Adapts
Choice
Granularity
Signal
Tensor core support
FourOverSix Cook et al. (2025)
scale
max. mapped to 4 or 6
16 elem.
local error
native
RaZeR Chen et al. (2025)
codebook
−0 code as block value
16 elem.
local error
emulated (two-pass GEMM)
IF4 Cook et al. (2026)
format
E2M1 or INT4
16 elem.
local error
new MAC unit needed
MixFP4 Zou et al. (2026)
format
E2M1 or E1M2
16 elem.
local error
modified core needed
BlockDialect Jang & Tambe (2025)
format
one of 16 FP4 variants
32 elem.
local error
new MAC unit needed
FlipQuant (ours)
format
E2M1 or E0M3
r×64 tile
end-to-end loss
native
Table 1: Block-scaled 4-bit quantization methods, by what they adapt (NVFP4 variants; BlockDialect builds on 32-element MX blocks). The granularity is the region that shares one adapted choice. Among the methods that adapt the element format, only FlipQuant runs natively on existing tensor cores and uses the end-to-end loss as its signal.
Figure 2: Overview of FlipQuant’s calibration. A binary map selects the fixed FourOverSix E2M1 or E0M3 candidate of each r×64 tile; scale blocks remain 1×16 . Tile gradients, the sums of G~⊙Δ over each tile, come from the bucketed KL divergence to a frozen BF16 teacher (Eq. ( 2 )). Adam updates only the latent parameters θ , whose hard threshold, with a straight-through estimator, gives the next map. k : calibration step; Attn, MLP: representative linear layers.
Figure 3: Extra GEMM time over the stock NVFP4 kernel for N×64 format tiles. (a) SM120, 40963 : a branch per MMA against our branch per K-tile (Appendix D.3 ). (b) SM100, 81923 on a GB200: one MMA of width N per tile (Appendix D.6 ; N=8 runs on one SM); log scale above 1%.
Figure 4: One warp’s K-tile on SM120, simplified to four of its 32 MMAs; widths are illustrative (Appendix D.3 measures (b)–(d)).
Llama-3.2
Llama-3.1
Mistral
Nemotron
Phi-4
Qwen3.8
PPL gap
1B
8B
7B
9B
14B
27B
closed
Method
Tile
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
(%) ↑
BF16
–
13.16
18.52
7.22
10.39
5.50
8.14
8.09
11.15
6.46
10.31
7.05
9.89
100.0
Uniform format, native on existing GPUs
NVFP4
–
15.54
21.77
7.89
11.34
5.73
8.42
8.45
11.54
6.69
10.58
7.57
10.22
0.0
FourOverSix
–
15.32
21.45
7.82
11.27
5.71
8.40
8.43
11.50
6.66
10.54
7.30
10.19
13.3
Table 2: W4A4 perplexity on WikiText-2 (Wiki) and C4 (lower is better). Bold and underlined: best and second-best natively executable quantized result per column (BF16 excluded; ties at the displayed precision share the mark). ‡ : GPTQ-quantized NVFP4 weights with FourOverSix activations. Last column: the share of NVFP4’s perplexity gap to BF16 that a method closes, 100⋅(PPLNVFP4−PPL)/(PPLNVFP4−PPLBF16) , averaged over the twelve model–corpus pairs.
Llama-3.2
Llama-3.1
Mistral
Nemotron
Phi-4
Qwen3.8
Mean
1B
8B
7B
9B
14B
27B
Method
Tile
7-task
GSM8K
7-task
GSM8K
7-task
GSM8K
7-task
GSM8K
7-task
GSM8K
7-task
GSM8K
7-task
GSM8K
BF16
–
57.95
45.4
72.94
85.9
74.06
47.0
72.67
87.4
74.50
93.3
75.20
95.7
71.22
75.8
Uniform format, native on existing GPUs
NVFP4
–
54.30
29.2
70.84
80.0
73.39
44.5
71.72
87.3
73.38
93.0
74.94
95.4
69.76
71.6
FourOverSix
–
54.11
29.7
70.88
79.5
73.20
44.5
71.20
88.9
73.37
91.7
74.86
94.6
69.60
71.5
Table 3: Downstream accuracy (%, higher is better). 7-task: mean over the seven log-likelihood tasks of Section 5.1 ; GSM8K: 8-shot chain of thought, strict match; Mean: over the six models. ‡ and marks as in Table 2 . Per-task results are in Appendix E .
Llama-3.2
Llama-3.1
Mistral
Nemotron
Phi-4
Qwen3.8
Method
1B
8B
7B
9B
14B
27B
RTX PRO 6000 (SM120): prefill, 1×2048 tokens, ms
BF16
22.9
131.2
125.1
151.1
244.7
453.7
NVFP4
12.0
61.3
55.6
79.8
109.9
201.7
FourOverSix
12.0
61.6
55.9
80.7
110.5
202.7
FlipQuant 16×64
12.2
61.7
56.0
81.4
111.8
204.5
Table 4: End-to-end prefill latency.
Map
8×64
16×64
256×64
Nemotron-9B
Random
−0.7
−0.3
+0.7
Activation-weighted
−1.3
−0.8
+2.8
One-shot gradient
+89.9
+56.8
−2.1
FlipQuant
−12.9
−11.9
−9.6
Qwen3.8-27B
Table 5: How the map is chosen: paired Δ NLL against FourOverSix on WikiText-2 ( ×10−3 nats per token, negative is better; 2 SE 0.9–8.6). Bold: best per column and maps within 2 SE of it.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Hugging Face ID
Type
Layers
Weights (B)
Tiles (M)
Llama-1B
meta-llama/Llama-3.2-1B-Instruct
Transformer
112
0.97
0.95
Llama-8B
meta-llama/Llama-3.1-8B-Instruct
Transformer
224
6.98
6.82
Mistral-7B
mistralai/Mistral-7B-Instruct-v0.3
Transformer
224
6.98
6.82
Nemotron-9B
nvidia/NVIDIA-Nemotron-Nano-9B-v2
Mamba-2 hybrid
120
7.71
7.53
Phi-4
microsoft/phi-4
Transformer
160
13.63
13.31
Qwen3.8-27B
Qwen/Qwen3.8-27B
DeltaNet hybrid
496
24.35
23.78
Appendix
Table A.1: Models. Type: DeltaNet hybrid = Gated DeltaNet and attention. Layers: the quantized linear layers (W4A4). Weights: the quantized weights in billions. Tiles: format tiles at 16×64 in millions; 8×64 has twice as many, and 256×64 one sixteenth, rounded up per matrix (Appendix D.1 ).
Model
Tile
WikiText-2
C4
Llama-1B
8×64
+0.06±1.41
−0.94±1.22
16×64
−0.71±1.85
−0.29±1.21
256×64
+0.15±1.76
+0.17±1.19
Llama-8B
8×64
−0.81±1.29
−0.87±1.24
16×64
+0.17±1.28
−0.75±1.08
256×64
+0.44±1.40
+0.21±1.33
Appendix
Table A.2: Native minus simulated evaluation of the FlipQuant maps of Table 2 on the RTX PRO 6000: paired Δ NLL ( ×10−3 nats per token, ± 2 SE). § : ∣ΔNLL∣ exceeds 2 SE (1 of 30 cells).
WikiText-2
C4
Model
Method
1×16
8×64
16×64
256×64
1×16
8×64
16×64
256×64
Llama-1B
IF4
15.34
15.97
16.03
16.01
21.07
22.35
22.24
22.39
MixFP4
15.25
15.93
16.07
16.05
21.05
22.20
22.21
22.26
FlipQuant
–
14.40
14.39
14.70
–
19.57
19.61
20.04
Reference
BF16 13.16, FourOverSix 15.32, NVFP4 15.54
BF16 18.52, FourOverSix 21.45, NVFP4 21.77
Llama-8B
IF4
7.74
7.82
7.82
7.84
11.18
11.28
11.30
11.31
Appendix
Table B.1: Perplexity of the per-block rules at 1×16 and at hardware tiles, and of FlipQuant at the same tiles (lower is better).
Mixed tiles (%)
Minority share (%)
Model
Rule
8×64
16×64
256×64
8×64
16×64
256×64
Llama-1B
IF4
100.0000
100.0000
100.0000
37.9
38.5
38.7
MixFP4
100.0000
100.0000
100.0000
37.8
38.4
38.6
Llama-8B
IF4
99.9999
100.0000
100.0000
37.3
37.8
38.0
MixFP4
99.9999
100.0000
100.0000
37.2
37.7
37.8
Mistral-7B
IF4
99.9996
99.9999
100.0000
37.4
38.0
38.1
Appendix
Table B.2: Mixing of each rule’s 1×16 choices inside the hardware tiles. Mixed tiles: the share (%) of tiles whose blocks choose both formats, truncated to four decimals. Minority share: the mean share (%) of the less frequent format among the blocks of a tile.
Setting
Value
Candidates
FourOverSix E2M1 and E0M3, as defined in Section 3.1
one FP32 latent parameter per tile; forward 1[θ>0] , backward ∂m/∂θ:=1 (Eq. ( 7 ))
Initialization
θ=−1 (the all-E2M1 FourOverSix map)
Optimizer
Adam, β=(0.9,0.999) , ϵ=10−12 , no weight decay
Step size
η=0.02 , constant, no warm-up or decay
Appendix
Table C.1: Calibration settings of every FlipQuant map in the paper.
Model
Setup (min)
Training (min)
Total (min)
GPU (GiB)
Llama-1B
0.9
2.6
3.7
16.6
Llama-8B
1.8
13.3
15.4
43.9
Mistral-7B
1.7
12.7
14.6
38.8
Nemotron-9B
11.2
159.0
170.4
55.9
Phi-4
3.0
25.0
28.1
62.1
Qwen3.8-27B
13.7
144.0
158.0
90.5
Appendix
Table C.2: Calibration cost of FlipQuant at 16×64 on one RTX PRO 6000. Setup: model load, calibration data, teacher log-probabilities, and candidate packing. Training: the five epochs (640 steps) together. Total: the whole calibration, from launch to the written map. GPU: peak allocated memory. Qwen3.8-27B keeps the teacher’s log-probabilities in host memory.
Time (s)
Model
8×64
16×64
256×64
Max. diff. (%)
Llama-1B
211
213
209
1.9
Llama-8B
901
910
915
1.5
Mistral-7B
858
864
865
0.8
Nemotron-9B
10,163
10,211
10,217
0.5
Phi-4
1,661
1,677
1,655
1.4
Appendix
Table C.3: Calibration time of FlipQuant at the three tiles on one RTX PRO 6000: the trainer’s time, Setup plus Training of Table C.2 , in seconds. The Total of Table C.2 adds 11–13 s for starting the trainer and checking and writing the map. Max. diff.: the largest difference among the three tiles, relative to the smallest time.
Tiles flipped to E0M3 (%)
Model
8×64
16×64
256×64
Llama-1B
11.28
14.84
39.69
Llama-8B
4.30
5.54
15.05
Mistral-7B
4.09
5.28
14.81
Nemotron-9B
4.29
5.59
15.98
Phi-4
2.85
3.69
10.32
Appendix
Table C.4: Tiles flipped from E2M1 to E0M3 by the learned maps, in percent of all tiles of the quantized layers.
Calibration
WikiText-2
C4
E0M3 (%)
Paper’s map
0
0
5.28
Seed 1
+0.44±0.95
−0.02±0.77
5.36
Seed 2
+0.15±0.98
+0.38±0.76
5.28
Non-deterministic
−0.30±0.93
−0.26±0.71
5.30
Web text
+0.28±1.02
−0.39±0.73
5.47
Instruction data
+2.30±1.02 ∗
+0.80±0.89
5.66
Appendix
Table C.5: Calibration choices on Mistral-7B at 16×64 : paired Δ NLL against the paper’s map ( ×10−3 nats per token, ± 2 SE; ∗ : beyond 2 SE; negative is better), and the percentage of tiles flipped to E0M3.
Epochs
Sequences
1
2
5
10
20
128
= FO6
= FO6
+2.3 ∗ / +1.6 ∗
−0.2 / −0.6
−2.3 ∗ / −2.9 ∗
256
= FO6
+5.5 ∗ / +4.0 ∗
0 / 0
−2.8 ∗ / −2.5 ∗
−2.9 ∗ / −3.7 ∗
512
+4.9 ∗ / +4.3 ∗
+1.4 ∗ / +0.2
−2.7 ∗ / −1.7 ∗
−3.1 ∗ / −2.8 ∗
−2.7 ∗ / −2.7 ∗
1,024
+1.0 / +0.5
−1.7 ∗ / −2.0 ∗
−3.1 ∗ / −2.4 ∗
−3.5 ∗ / −3.8 ∗
−3.4 ∗ / −4.6 ∗
2,048
−1.3 ∗ / −1.8 ∗
−3.2 ∗ / −2.7 ∗
−4.5 ∗ / −3.7 ∗
−5.7 ∗ / −4.4 ∗
−4.0 ∗ / −4.6 ∗
Appendix
Table C.6: Calibration budget on Mistral-7B at 16×64 : paired Δ NLL against a reference map of 256 sequences and 5 epochs (0 / 0) on WikiText-2 / C4 ( ×10−3 nats per token; ∗ : beyond 2 SE, which is 0.8–1.8; negative is better). Bold: the paper’s setting, 1,024 sequences for 5 epochs. Steps: sequences × epochs /8 . = FO6: no tile is flipped, and the map is FourOverSix ( +9.7 / +9.0 against the reference).
Epochs
Sequences
1
2
5
10
20
128
0.00
0.00
0.24
1.84
3.52
256
0.00
0.04
1.11
3.00
6.00
512
0.04
0.29
2.48
5.68
11.16
1,024
0.22
1.07
5.28
10.90
20.09
2,048
0.66
3.35
10.67
19.69
31.45
Appendix
Table C.7: Tiles flipped to E0M3 (%) by the maps of Table C.6 ; the paper’s setting in bold, and the reference map is 256 sequences for 5 epochs.
SM120 (RTX PRO 6000, RTX 5090)
SM100 (B200, GB200)
Instruction
mma.sync m16n8k64 , mxf4nvf4
tcgen05.mma , mxf4nvf4
Format field
opcode
instruction descriptor, in a register
Weight operand
A ( 16×64 , 256×64 ) or B ( 8×64 )
B
Selecting E0M3
branch per K-tile into an arm
register write per MMA
Map storage
bit 7 of the tile’s scale bytes
one bit per tile
Toolchain
post-compile opcode patch
descriptor override
Appendix
Table D.1: How a format tile is executed on the two Blackwell variants. Activations are E2M1 on both.
Tile
Branch
μ s
TFLOP/s
Iso.
B2B
MMAs
RTX PRO 6000
Stock NVFP4
118.48
1160.0
–
–
32
8×64
per MMA
184.58
744.6
+55.8%
+69.9%
64
per K-tile
125.99
1090.9
+6.3%
+7.8%
32
16×64
per MMA
177.48
774.4
+49.8%
+64.3%
64
per K-tile
119.96
1145.7
+1.2%
+3.8%
32
Appendix
Table D.2: Branch per MMA against branch per K-tile at 40963 , with the tags of a real layer. μ s and TFLOP/s: GEMM time and throughput, isolated launches with cold weights. Iso. and B2B: extra time over the stock NVFP4 kernel, isolated and back-to-back launches. MMAs: tensor-pipe instructions per warp and K-tile. RTX PRO 6000: mean of 10 repeats in the session of Table D.3 , with its tags (Mistral-7B); RTX 5090: tags of a Llama-3.1-8B map calibrated on that GPU. With all tiles E2M1, the per-MMA overheads change by at most 0.2 points.
Part
8×64
16×64
256×64
Before optimization
Tiles
+3.6±0.2
+1.1±0.2
+1.1±0.2
Tag copies
+0.9±0.1
+0.6±0.2
+0.6±0.2
Dispatch
+3.9±0.3
+1.8±0.3
+1.8±0.3
E0M3 tiles
+0.6±0.6
+0.3±0.4
+0.9±0.4
Total
+9.2±0.5
+3.9±0.4
+4.5±0.3
Appendix
Table D.3: Overhead over the deployed stock kernel at 40963 (%), split into its parts, before and after the optimizations of this section. Isolated launches with cold weights, as in Table D.2 : mean ± standard deviation over 10 repeats; all kernels interleaved in one session on one RTX PRO 6000. Tags of layer 0 o_proj of Mistral-7B-Instruct-v0.3 from its maps of Table 2 . The first 256×64 kernel is the 16-arm 16×64 kernel, whose tiles and tag copies it shares.
Tiles flipped to E0M3 (%)
Tile
T
0
1
2
5
10
25
50
100
8×64
16
2.1
2.2
2.4
3.1
3.9
4.5
5.7
5.8
128
1.9
1.8
2.0
2.5
2.9
3.5
4.1
5.1
512
8.6
9.4
10.4
11.7
13.4
17.2
20.7
20.5
2048
8.9
9.3
9.4
10.4
11.6
13.9
15.6
15.8
16×64
16
0.7
1.0
0.8
1.3
1.9
1.9
2.6
2.4
Appendix
Table D.4: Overhead over the stock kernel (%) against the tiles flipped to E0M3 (%), at random positions, for the deployed kernels. Mean over the o_proj ( 4096×4096 ), gate_proj ( 14336×4096 ) and down_proj ( 4096×14336 ) shapes of Mistral-7B-v0.3, with random weights; each cell is the median of 90 launches.
T
8×64
16×64
256×64
1
+0.65 ( +0.80 )
+0.58 ( +0.94 )
+0.12 ( +0.25 )
4
+0.12 ( +0.39 )
+0.44 ( +0.81 )
+0.04 ( +0.04 )
16
+0.96 ( +1.48 )
+0.94 ( +1.12 )
+0.20 ( +0.30 )
32
+2.45 ( +3.33 )
+1.15 ( +1.43 )
+0.39 ( +0.38 )
64
+1.29 ( +1.55 )
+1.50 ( +2.17 )
+0.32 ( +0.39 )
128
+2.18 ( +2.92 )
+1.67 ( +2.25 )
+0.53 ( +0.54 )
Appendix
Table D.5: Overhead of the GEMMs of a forward pass of the deployed kernels over the stock kernel (%), median over the six models with their maps of Table 2 , on one RTX PRO 6000 (driver 595.71, 500 W). Tags of each projection: from its module with the median E0M3 share, and from its module with the most E0M3 tiles in parentheses. Last row: the median over all (model, T ) cells with T≥128 .
Method
Tile
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
LAMBADA
MMLU
Avg.
Llama-1B
BF16
–
75.14
61.67
61.48
63.80
37.97
59.46
46.12
57.95
NVFP4
–
71.65
57.72
59.91
59.97
36.26
53.10
41.47
54.30
FourOverSix
–
71.38
57.74
58.17
60.19
36.35
53.64
41.34
54.11
GPTQ ‡
–
72.09
57.79
59.67
60.19
35.58
53.93
42.37
54.52
IF4 †
1×16
71.93
58.31
60.77
60.52
36.18
54.86
41.24
54.83
Appendix
Table E.1: Multiple-choice accuracy (%, higher is better) on Llama-1B and Llama-8B. PIQA, HellaSwag, and ARC (ARC-e, ARC-c): length-normalized accuracy; the other tasks: accuracy. MMLU 5-shot, the other tasks 0-shot (Appendix A.3 ). Avg.: mean over the seven tasks, the value of Table 3 . ‡ : GPTQ-quantized NVFP4 weights with FourOverSix activations; † : per-block format, simulated. Bold and underlined: best and second-best quantized result per column and model (BF16 excluded; ties share the mark).
Method
Tile
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
LAMBADA
MMLU
Avg.
Mistral-7B
BF16
–
83.51
83.32
74.59
81.73
59.90
73.53
61.84
74.06
NVFP4
–
82.86
82.73
74.03
81.78
59.13
73.24
59.95
73.39
FourOverSix
–
82.75
82.86
74.82
80.64
58.36
73.12
59.87
73.20
GPTQ ‡
–
83.46
82.81
74.03
81.44
59.04
72.89
60.25
73.42
IF4 †
1×16
83.19
82.90
75.37
80.98
58.11
72.77
60.26
73.37
Appendix
Table E.2: Multiple-choice accuracy (%, higher is better) on Mistral-7B and Nemotron-9B; metrics, Avg., and marks as in Table E.1 .
Method
Tile
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
LAMBADA
MMLU
Avg.
Phi-4
BF16
–
81.12
81.95
76.95
72.77
56.14
72.27
80.29
74.50
NVFP4
–
80.85
80.86
73.64
73.23
54.10
71.88
79.11
73.38
FourOverSix
–
80.69
81.11
74.27
71.89
54.86
71.55
79.23
73.37
GPTQ ‡
–
80.74
81.17
73.95
72.90
55.55
72.75
79.27
73.76
IF4 †
1×16
81.01
81.15
74.82
72.52
55.38
71.49
79.45
73.69
Appendix
Table E.3: Multiple-choice accuracy (%, higher is better) on Phi-4 and Qwen3.8-27B; metrics, Avg., and marks as in Table E.1 .
Figure F.1: The tile gradient against the measured change in bucketed KL divergence on Mistral-7B at 16×64 . Measured: the change in the KL divergence from the BF16 teacher when the tiles are flipped from the all-E2M1 map, paired over the sequences of each set; predicted: the sum of the tiles’ gu , its first-order estimate; negative means the KL divergence falls. (a) The 30 groups of 1,000 tiles: measured change ( ± 2 SE) against the predicted change ∑gu , colored by the bin of gu each group is drawn from (dashed: measured Δ KL equals predicted Δ KL; the horizontal axis is linear between −0.1 and 0.1 and logarithmic beyond). (b) The k tiles with the most negative gu flipped together. Top: measured ( ± 2 SE) and predicted change (dashed; the triangle marks its value at 359,840 tiles, which lies below the axis); stars: the paper’s map, whose 359,840 tiles were chosen by calibration (dotted line: its size; horizontal dashed lines: its measured changes; hollow: its predicted change). Bottom: measured divided by predicted change (dashed: equal).
Score
Held-out
In-sample
gu
+0.72 [ +0.40 , +0.90 ]
+0.88 [ +0.69 , +0.97 ]
Act.-weighted
−0.43 [ −0.76 , −0.01 ]
−0.46 [ −0.78 , −0.04 ]
Weight error
−0.27 [ −0.69 , +0.18 ]
−0.19 [ −0.62 , +0.27 ]
Appendix
Table F.1: Ranking groups of tiles: Spearman correlation between each score and the measured decrease in bucketed KL divergence over the 30 groups of 1,000 tiles (Figure F.1 a); 95% bootstrap interval over groups in brackets. Act.-weighted: the activation-weighted score of Table 5 .
Nemotron-9B
Qwen3.8-27B
PTQ
Format
Wiki
C4
GSM8K
Wiki
C4
GSM8K
BF16
8.09
11.15
87.4
7.05
9.89
95.7
RTN
NVFP4
8.45
11.54
87.3
7.57
10.22
95.4
FourOverSix
8.43
11.50
88.9
7.30
10.19
94.6
FlipQuant
8.33
11.44
88.6
7.08
10.12
95.9
GPTQ
NVFP4
8.43
11.50
84.5
7.44
10.17
95.9
Appendix
Table F.2: Combining FlipQuant ( 16×64 ) with post-training quantization methods, native execution: WikiText-2 and C4 perplexity (lower is better) and GSM8K accuracy (%, higher is better). GPTQ compensates the quantization error with second-order updates, with activations quantized during calibration; its FlipQuant row trains the map on GPTQ-quantized candidates of both formats. Hadamard rotates weights and activations in blocks of 16 before quantization, and its FlipQuant map is trained in the rotated basis. Format sets both the weights and the activations: NVFP4 rows use NVFP4 activations, the others FourOverSix activations. The GPTQ baseline of Table 2 instead quantizes standard NVFP4 weights with GPTQ calibrated on BF16 activations and uses FourOverSix activations. GSM8K follows lm-eval’s gsm8k_llama (8-shot chain of thought with the chat template, greedy decoding, at most 1024 new tokens, thinking off, strict match); its 2 SE is 1.0–2.0 points. Bold: within each PTQ method, the lowest perplexity and the highest GSM8K accuracy per column; ties are all bold.
Four-bit floating-point (FP4) Tensor Cores accelerate matrix multiplication, but scale computation, operand packing, layout construction, and saved backward state can erase the gain. We present format-aware fusion, which co-designs each quantization producer with its scale domain and consumer layout for native \mxfp{}, global \nvfp{}, and cooperative-thread-array-local \nvfp{}. We evaluate Llama-3-family 8B pretraining through 160 billion tokens using bfloat16 output projections and compiled cross entropy. In matched same-accelerator probes, bfloat16 and Transformer Engine \nvfp{} reach 18.8K and 27.6K tokens/s/GPU, while our fastest custom route reaches 37.9K. \mxfp{} with row-gradient stochastic rounding and fixed-sign 32-value Hadamard weight-gradient preconditioning reaches 37.2K tokens/s/GPU (86.3% bfloat16 model FLOP utilization) and ends 2.11% above the raw bfloat16 training-loss endpoint. A Transformer Engine recipe with four final bfloat16 blocks ends 0.87% above bfloat16 at 27.1K tokens/s/GPU. Downstream rankings differ from training-loss rankings, showing that FP4 outcomes depend jointly on scale contract, operand, and execution path.
Large language models make weight storage and memory traffic major inference costs, motivating low-precision formats that represent each weight with only a few bits. Such formats use a scale to map floating-point values into a small codebook; NVFP4 improves local range utilization by letting every 16 E2M1 weights share an E4M3 block scale. Choosing that scale is difficult in GPTQ because quantizing one column updates those that follow, so evaluating a block independently can misestimate its final reconstruction error. Large models pose a second challenge: full-precision weights, calibration activations, and second-order state cannot all remain on one accelerator, while assigning complete layers to devices leaves each time-consuming layer solve serial. We introduce Schur Replay, a scale-selection algorithm that reproduces the GPTQ updates caused by each block scale and scores the resulting block error after accounting for compensation from unquantized columns. Separately, our execution infrastructure keeps only the active layer resident, tiers activations across device, host, and disk, retires full-precision layers after export, and distributes independent output rows across tensor-parallel ranks. Together, the algorithm and infrastructure attain 99.35% and 100.84% question-weighted recovery from BF16 across seven benchmarks on Qwen3.5-397B-A17B and Llama-3.3-70B-Instruct. On the 397B model, the infrastructure reduces measured per-layer time by 15.17× over ModelOpt and 23.14× over LLM Compressor, with lower memory used per GPU.
Ruiyi Ding, Jie Li, Kang He +4
KlingAI Research · University of Science and Technology of China · Fudan University +2
Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies dominate once its matrix products shrink. We address this with Direct-P for noncausal inference and a causal path that passes the forward quantization directly into backward. Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13× the bfloat16 (BF16) forward throughput on an NVIDIA GB200. The causal path reconstructs probabilities from saved quantized queries and keys and uses 8-bit floating-point (FP8) gradient operands, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14×. Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.