Gradient-Guided Hardware-Native MIXFP4
Organizations: National Yang Ming Chiao Tung University, Hsinchu, Taiwan · National Taiwan University, Taipei, Taiwan · OpenAI, San Francisco, CA, USA · Cornell University, USA
Abstract
Blackwell tensor cores support native inference with 4-bit weights and activations (W4A4), but NVFP4 leaves a perplexity (PPL) gap to BF16. Choosing a format for each 16-element block can improve accuracy, but current matrix multiply instructions cannot execute choices this fine. Applying these rules at hardware-compatible tiles loses most of their benefit. We introduce FlipQuant, which selects E2M1 or E0M3 for each MMA weight tile. It keeps both quantized weight candidates fixed and learns a binary format map from end-to-end gradients of a bucketed KL loss to a BF16 teacher, re-evaluating them after each update. Native kernels execute 16 x 64 tiles on SM120 and 256 x 64 tiles on SM100 with median GEMM overheads of 1.9% and 1.7%, respectively. Across six language models and two corpora, FlipQuant at 16 x 64 closes 42.5% of NVFP4's PPL gap to BF16 on average, compared with 13.3% for FourOverSix. Code is available at https://github.com/brian030128/flipquant.
Figures & tables
| Method | Adapts | Choice | Granularity | Signal | Tensor core support |
|---|---|---|---|---|---|
| FourOverSix Cook et al. (2025) | scale | max. mapped to 4 or 6 | 16 elem. | local error | native |
| RaZeR Chen et al. (2025) | codebook | code as block value | 16 elem. | local error | emulated (two-pass GEMM) |
| IF4 Cook et al. (2026) | format | E2M1 or INT4 | 16 elem. | local error | new MAC unit needed |
| MixFP4 Zou et al. (2026) | format | E2M1 or E1M2 | 16 elem. | local error | modified core needed |
| BlockDialect Jang & Tambe (2025) | format | one of 16 FP4 variants | 32 elem. | local error | new MAC unit needed |
| FlipQuant (ours) | format | E2M1 or E0M3 | tile | end-to-end loss | native |
| Llama-3.2 | Llama-3.1 | Mistral | Nemotron | Phi-4 | Qwen3.8 | PPL gap | ||||||||
| 1B | 8B | 7B | 9B | 14B | 27B | closed | ||||||||
| Method | Tile | Wiki | C4 | Wiki | C4 | Wiki | C4 | Wiki | C4 | Wiki | C4 | Wiki | C4 | (%) |
| BF16 | – | 13.16 | 18.52 | 7.22 | 10.39 | 5.50 | 8.14 | 8.09 | 11.15 | 6.46 | 10.31 | 7.05 | 9.89 | 100.0 |
| Uniform format, native on existing GPUs | ||||||||||||||
| NVFP4 | – | 15.54 | 21.77 | 7.89 | 11.34 | 5.73 | 8.42 | 8.45 | 11.54 | 6.69 | 10.58 | 7.57 | 10.22 | 0.0 |
| FourOverSix | – | 15.32 | 21.45 | 7.82 | 11.27 | 5.71 | 8.40 | 8.43 | 11.50 | 6.66 | 10.54 | 7.30 | 10.19 | 13.3 |
| Llama-3.2 | Llama-3.1 | Mistral | Nemotron | Phi-4 | Qwen3.8 | Mean | |||||||||
| 1B | 8B | 7B | 9B | 14B | 27B | ||||||||||
| Method | Tile | 7-task | GSM8K | 7-task | GSM8K | 7-task | GSM8K | 7-task | GSM8K | 7-task | GSM8K | 7-task | GSM8K | 7-task | GSM8K |
| BF16 | – | 57.95 | 45.4 | 72.94 | 85.9 | 74.06 | 47.0 | 72.67 | 87.4 | 74.50 | 93.3 | 75.20 | 95.7 | 71.22 | 75.8 |
| Uniform format, native on existing GPUs | |||||||||||||||
| NVFP4 | – | 54.30 | 29.2 | 70.84 | 80.0 | 73.39 | 44.5 | 71.72 | 87.3 | 73.38 | 93.0 | 74.94 | 95.4 | 69.76 | 71.6 |
| FourOverSix | – | 54.11 | 29.7 | 70.88 | 79.5 | 73.20 | 44.5 | 71.20 | 88.9 | 73.37 | 91.7 | 74.86 | 94.6 | 69.60 | 71.5 |
| Llama-3.2 | Llama-3.1 | Mistral | Nemotron | Phi-4 | Qwen3.8 | |
| Method | 1B | 8B | 7B | 9B | 14B | 27B |
| RTX PRO 6000 (SM120): prefill, tokens, ms | ||||||
| BF16 | 22.9 | 131.2 | 125.1 | 151.1 | 244.7 | 453.7 |
| NVFP4 | 12.0 | 61.3 | 55.6 | 79.8 | 109.9 | 201.7 |
| FourOverSix | 12.0 | 61.6 | 55.9 | 80.7 | 110.5 | 202.7 |
| FlipQuant | 12.2 | 61.7 | 56.0 | 81.4 | 111.8 | 204.5 |
| Map | |||
|---|---|---|---|
| Nemotron-9B | |||
| Random | |||
| Activation-weighted | |||
| One-shot gradient | |||
| FlipQuant | |||
| Qwen3.8-27B | |||
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Hugging Face ID | Type | Layers | Weights (B) | Tiles (M) |
|---|---|---|---|---|---|
| Llama-1B | meta-llama/Llama-3.2-1B-Instruct | Transformer | 112 | 0.97 | 0.95 |
| Llama-8B | meta-llama/Llama-3.1-8B-Instruct | Transformer | 224 | 6.98 | 6.82 |
| Mistral-7B | mistralai/Mistral-7B-Instruct-v0.3 | Transformer | 224 | 6.98 | 6.82 |
| Nemotron-9B | nvidia/NVIDIA-Nemotron-Nano-9B-v2 | Mamba-2 hybrid | 120 | 7.71 | 7.53 |
| Phi-4 | microsoft/phi-4 | Transformer | 160 | 13.63 | 13.31 |
| Qwen3.8-27B | Qwen/Qwen3.8-27B | DeltaNet hybrid | 496 | 24.35 | 23.78 |
| Model | Tile | WikiText-2 | C4 |
|---|---|---|---|
| Llama-1B | |||
| Llama-8B | |||
| WikiText-2 | C4 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Method | ||||||||
| Llama-1B | IF4 | 15.34 | 15.97 | 16.03 | 16.01 | 21.07 | 22.35 | 22.24 | 22.39 |
| MixFP4 | 15.25 | 15.93 | 16.07 | 16.05 | 21.05 | 22.20 | 22.21 | 22.26 | |
| FlipQuant | – | 14.40 | 14.39 | 14.70 | – | 19.57 | 19.61 | 20.04 | |
| Reference | BF16 13.16, FourOverSix 15.32, NVFP4 15.54 | BF16 18.52, FourOverSix 21.45, NVFP4 21.77 | |||||||
| Llama-8B | IF4 | 7.74 | 7.82 | 7.82 | 7.84 | 11.18 | 11.28 | 11.30 | 11.31 |
| Mixed tiles (%) | Minority share (%) | ||||||
|---|---|---|---|---|---|---|---|
| Model | Rule | ||||||
| Llama-1B | IF4 | 100.0000 | 100.0000 | 100.0000 | 37.9 | 38.5 | 38.7 |
| MixFP4 | 100.0000 | 100.0000 | 100.0000 | 37.8 | 38.4 | 38.6 | |
| Llama-8B | IF4 | 99.9999 | 100.0000 | 100.0000 | 37.3 | 37.8 | 38.0 |
| MixFP4 | 99.9999 | 100.0000 | 100.0000 | 37.2 | 37.7 | 37.8 | |
| Mistral-7B | IF4 | 99.9996 | 99.9999 | 100.0000 | 37.4 | 38.0 | 38.1 |
| Setting | Value |
|---|---|
| Candidates | FourOverSix E2M1 and E0M3, as defined in Section 3.1 |
| Activations | per-token FourOverSix; straight-through estimator (Eq. ( 4 )) |
| Map parameters | one FP32 latent parameter per tile; forward , backward (Eq. ( 7 )) |
| Initialization | (the all-E2M1 FourOverSix map) |
| Optimizer | Adam, , , no weight decay |
| Step size | , constant, no warm-up or decay |
| Model | Setup (min) | Training (min) | Total (min) | GPU (GiB) |
|---|---|---|---|---|
| Llama-1B | 0.9 | 2.6 | 3.7 | 16.6 |
| Llama-8B | 1.8 | 13.3 | 15.4 | 43.9 |
| Mistral-7B | 1.7 | 12.7 | 14.6 | 38.8 |
| Nemotron-9B | 11.2 | 159.0 | 170.4 | 55.9 |
| Phi-4 | 3.0 | 25.0 | 28.1 | 62.1 |
| Qwen3.8-27B | 13.7 | 144.0 | 158.0 | 90.5 |
| Time (s) | ||||
| Model | Max. diff. (%) | |||
| Llama-1B | 211 | 213 | 209 | 1.9 |
| Llama-8B | 901 | 910 | 915 | 1.5 |
| Mistral-7B | 858 | 864 | 865 | 0.8 |
| Nemotron-9B | 10,163 | 10,211 | 10,217 | 0.5 |
| Phi-4 | 1,661 | 1,677 | 1,655 | 1.4 |
| Tiles flipped to E0M3 (%) | |||
|---|---|---|---|
| Model | |||
| Llama-1B | 11.28 | 14.84 | 39.69 |
| Llama-8B | 4.30 | 5.54 | 15.05 |
| Mistral-7B | 4.09 | 5.28 | 14.81 |
| Nemotron-9B | 4.29 | 5.59 | 15.98 |
| Phi-4 | 2.85 | 3.69 | 10.32 |
| Calibration | WikiText-2 | C4 | E0M3 (%) |
|---|---|---|---|
| Paper’s map | 0 | 0 | 5.28 |
| Seed 1 | 5.36 | ||
| Seed 2 | 5.28 | ||
| Non-deterministic | 5.30 | ||
| Web text | 5.47 | ||
| Instruction data | ∗ | 5.66 |
| Epochs | |||||
|---|---|---|---|---|---|
| Sequences | 1 | 2 | 5 | 10 | 20 |
| 128 | = FO6 | = FO6 | ∗ / ∗ | / | ∗ / ∗ |
| 256 | = FO6 | ∗ / ∗ | 0 / 0 | ∗ / ∗ | ∗ / ∗ |
| 512 | ∗ / ∗ | ∗ / | ∗ / ∗ | ∗ / ∗ | ∗ / ∗ |
| 1,024 | / | ∗ / ∗ | ∗ / ∗ | ∗ / ∗ | ∗ / ∗ |
| 2,048 | ∗ / ∗ | ∗ / ∗ | ∗ / ∗ | ∗ / ∗ | ∗ / ∗ |
| Epochs | |||||
|---|---|---|---|---|---|
| Sequences | 1 | 2 | 5 | 10 | 20 |
| 128 | 0.00 | 0.00 | 0.24 | 1.84 | 3.52 |
| 256 | 0.00 | 0.04 | 1.11 | 3.00 | 6.00 |
| 512 | 0.04 | 0.29 | 2.48 | 5.68 | 11.16 |
| 1,024 | 0.22 | 1.07 | 5.28 | 10.90 | 20.09 |
| 2,048 | 0.66 | 3.35 | 10.67 | 19.69 | 31.45 |
| SM120 (RTX PRO 6000, RTX 5090) | SM100 (B200, GB200) | |
|---|---|---|
| Instruction | mma.sync m16n8k64 , mxf4nvf4 | tcgen05.mma , mxf4nvf4 |
| Format field | opcode | instruction descriptor, in a register |
| Weight operand | ( , ) or ( ) | |
| Selecting E0M3 | branch per K-tile into an arm | register write per MMA |
| Map storage | bit 7 of the tile’s scale bytes | one bit per tile |
| Toolchain | post-compile opcode patch | descriptor override |
| Tile | Branch | s | TFLOP/s | Iso. | B2B | MMAs |
| RTX PRO 6000 | ||||||
| Stock NVFP4 | 118.48 | 1160.0 | – | – | 32 | |
| per MMA | 184.58 | 744.6 | +55.8% | +69.9% | 64 | |
| per K-tile | 125.99 | 1090.9 | +6.3% | +7.8% | 32 | |
| per MMA | 177.48 | 774.4 | +49.8% | +64.3% | 64 | |
| per K-tile | 119.96 | 1145.7 | +1.2% | +3.8% | 32 | |
| Part | |||
|---|---|---|---|
| Before optimization | |||
| Tiles | |||
| Tag copies | |||
| Dispatch | |||
| E0M3 tiles | |||
| Total | |||
| Tiles flipped to E0M3 (%) | |||||||||
| Tile | 0 | 1 | 2 | 5 | 10 | 25 | 50 | 100 | |
| 16 | 2.1 | 2.2 | 2.4 | 3.1 | 3.9 | 4.5 | 5.7 | 5.8 | |
| 128 | 1.9 | 1.8 | 2.0 | 2.5 | 2.9 | 3.5 | 4.1 | 5.1 | |
| 512 | 8.6 | 9.4 | 10.4 | 11.7 | 13.4 | 17.2 | 20.7 | 20.5 | |
| 2048 | 8.9 | 9.3 | 9.4 | 10.4 | 11.6 | 13.9 | 15.6 | 15.8 | |
| 16 | 0.7 | 1.0 | 0.8 | 1.3 | 1.9 | 1.9 | 2.6 | 2.4 | |
| 1 | ( ) | ( ) | ( ) |
|---|---|---|---|
| 4 | ( ) | ( ) | ( ) |
| 16 | ( ) | ( ) | ( ) |
| 32 | ( ) | ( ) | ( ) |
| 64 | ( ) | ( ) | ( ) |
| 128 | ( ) | ( ) | ( ) |
| Method | Tile | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | LAMBADA | MMLU | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Llama-1B | |||||||||
| BF16 | – | 75.14 | 61.67 | 61.48 | 63.80 | 37.97 | 59.46 | 46.12 | 57.95 |
| NVFP4 | – | 71.65 | 57.72 | 59.91 | 59.97 | 36.26 | 53.10 | 41.47 | 54.30 |
| FourOverSix | – | 71.38 | 57.74 | 58.17 | 60.19 | 36.35 | 53.64 | 41.34 | 54.11 |
| GPTQ ‡ | – | 72.09 | 57.79 | 59.67 | 60.19 | 35.58 | 53.93 | 42.37 | 54.52 |
| IF4 † | 71.93 | 58.31 | 60.77 | 60.52 | 36.18 | 54.86 | 41.24 | 54.83 | |
| Method | Tile | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | LAMBADA | MMLU | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Mistral-7B | |||||||||
| BF16 | – | 83.51 | 83.32 | 74.59 | 81.73 | 59.90 | 73.53 | 61.84 | 74.06 |
| NVFP4 | – | 82.86 | 82.73 | 74.03 | 81.78 | 59.13 | 73.24 | 59.95 | 73.39 |
| FourOverSix | – | 82.75 | 82.86 | 74.82 | 80.64 | 58.36 | 73.12 | 59.87 | 73.20 |
| GPTQ ‡ | – | 83.46 | 82.81 | 74.03 | 81.44 | 59.04 | 72.89 | 60.25 | 73.42 |
| IF4 † | 83.19 | 82.90 | 75.37 | 80.98 | 58.11 | 72.77 | 60.26 | 73.37 | |
| Method | Tile | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | LAMBADA | MMLU | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Phi-4 | |||||||||
| BF16 | – | 81.12 | 81.95 | 76.95 | 72.77 | 56.14 | 72.27 | 80.29 | 74.50 |
| NVFP4 | – | 80.85 | 80.86 | 73.64 | 73.23 | 54.10 | 71.88 | 79.11 | 73.38 |
| FourOverSix | – | 80.69 | 81.11 | 74.27 | 71.89 | 54.86 | 71.55 | 79.23 | 73.37 |
| GPTQ ‡ | – | 80.74 | 81.17 | 73.95 | 72.90 | 55.55 | 72.75 | 79.27 | 73.76 |
| IF4 † | 81.01 | 81.15 | 74.82 | 72.52 | 55.38 | 71.49 | 79.45 | 73.69 | |
| Score | Held-out | In-sample |
|---|---|---|
| [ , ] | [ , ] | |
| Act.-weighted | [ , ] | [ , ] |
| Weight error | [ , ] | [ , ] |
| Nemotron-9B | Qwen3.8-27B | ||||||
|---|---|---|---|---|---|---|---|
| PTQ | Format | Wiki | C4 | GSM8K | Wiki | C4 | GSM8K |
| BF16 | 8.09 | 11.15 | 87.4 | 7.05 | 9.89 | 95.7 | |
| RTN | NVFP4 | 8.45 | 11.54 | 87.3 | 7.57 | 10.22 | 95.4 |
| FourOverSix | 8.43 | 11.50 | 88.9 | 7.30 | 10.19 | 94.6 | |
| FlipQuant | 8.33 | 11.44 | 88.6 | 7.08 | 10.12 | 95.9 | |
| GPTQ | NVFP4 | 8.43 | 11.50 | 84.5 | 7.44 | 10.17 | 95.9 |