Integer-Exact Microscaling: Running MXFP4 Language Models on INT8 Matrix Units
Authors: Shinyoung Kang, Junhyung Ahn, Yujun Kim, Jungho Park
Organizations: UT Southwestern Medical Center · Department of Data and Decision Sciences, Emory University · Fuqua School of Business, Duke University · Heinz College of Information Systems and Public Policy, Carnegie Mellon University
On hardware whose fastest matrix datapath is INT8×INT8, such as the Neural Accelerators of Apple's M5 GPUs, 4-bit weights must become INT8 operands. Existing MXFP4 kernels already map FP4 (E2M1) values to integers by doubling them, but apply every 32-element block scale inside the reduction. We show when no per-block work is needed: a row of MXFP4 weights is exactly one INT8 vector times one power-of-two scale if its block exponents span at most three, so a linear layer becomes a single INT8 GEMM with one rescale per output. Enforcing the condition adds at most 1/16 of a row's largest rounding error to the blocks it changes, and real weights already satisfy it: 97.9-99.97% of rows across seven models, including 99.93% of the 95M rows of a 125B-parameter mixture-of-experts (MoE) model, Qwen3.8-Flash-Next. The resulting spread-constrained MXFP4 files stay standard MXFP4. With integer instead of power-of-two block multipliers, an integer-exact format (16-wide blocks, multipliers 1..10, 4.26 bits) runs as one exact INT8 GEMM and is more accurate than NVFP4 (4.5 bits) with its standard scale rule on five small models, within 0.06 perplexity of NVFP4 with scale search. On Qwen3.8-Flash-Next's routed experts with INT8 activations it is as close to the FP8 model as NVFP4 with scale search (KL divergence 0.0557 vs. 0.0553) and closer than MLX's MXFP4 and affine INT4; with every matrix quantized it solves 94.8% of GSM8K vs. 91.0% for MLX's MXFP4. Quantizing only the 74k routed expert matrices (96% of the parameters) to either of our formats with INT8 activations costs at most 0.2% perplexity. A complete W4A8 system on the M5 produces outputs bit-identical to the quantized reference; prefill is 1.34-1.74x faster than MLX on a dense model, 1.10-1.57x on two MoE models up to 35B parameters and 1.22-1.29x on Qwen3.8-Flash-Next, where shared non-GEMM work bounds the gain.
Figures & tables
Figure 1: Left: deployed 4-bit formats are dequantized to FP16 and multiplied at the FP16 rate (W4A16). We rewrite MXFP4 weights exactly as INT8 operands and quantize activations to INT8, so the GEMM runs on the INT8 datapath, about 2× faster on Apple M5. The weights remain a valid MXFP4 file. Right: the rewrite is exact when a row’s block exponents lie within a window of four values below the largest; blocks below it (red) are coarsened, which adds at most 1/16 of the row’s largest rounding error (Proposition 2 ).
Figure 2: matmul2d throughput on the M5 Max Neural Accelerators by operand type (left × right operand) for M×N×K shapes. INT8 × INT8 reaches 1.8 – 2.0× FP16; every mixed float × integer combination is slower than FP16.
Format
Bits
Output error ( ×103 )
INT8-exact
NVFP4, scale search
4.50
5.70
no
NVFP4 (amax/6 rule)
4.50
6.02
no
E2M1, 16-blocks, m∈{1..10}
4.26
7.36
yes
E2M1, 16-blocks, m∈{1..8}
4.19
7.97
yes
E2M1, 32-blocks, m∈{1..8}
4.10
8.88
yes
E2M1, 32-blocks, m∈{1..10}
4.13
8.99
yes
Table 1: Integer-exact formats (Eq. 3 ) vs. standard 4-bit formats on Qwen3-1.7B: layer-output error tr(ΔWHΔW⊤)/tr(WHW⊤) with H from WikiText-2 train activations, every third layer, round-to-nearest. Bits include block codes and one FP32 scale per row ( K=4096 ). “+ rotation”: randomized Hadamard transform of each 32-wide input block. Scale search: each NVFP4 block’s E4M3 scale and its two neighbours on either side; exponent search: the OCP exponent and its two neighbours (least squared weight error).
Qwen3
SmolLM3
OLMoE
Qwen3.5
Method
Bits
INT8
0.6B
1.7B
4B
3B
1B-7B ‡
35B-A3B ‡
BF16
16
12.67
9.40
7.90
7.91
7.49
5.72
INT4 affine g64 (MLX) W4A16
4.50
no
15.83
12.10
8.56
8.44
7.86
5.92
NVFP4 W4A16
4.50
no
14.44
10.23
8.26
8.29
7.82
5.87
NVFP4 W4A16, scale search
4.50
no
–
10.08
–
–
–
–
NVFP4 W4A8
4.50
no
14.91
10.51
8.47
8.36
7.92
5.93
Table 2: WikiText-2 perplexity. W4A16: 4-bit weights, 16-bit activations; W4A8: per-token INT8 activations. SC-MXFP4 W4A8 is what our kernels compute. All methods quantize the same matrices. ‡ MoE: every expert is quantized, the router stays BF16. Bits at K=4096 (the FP32 row scales of our formats add 32/K ). – = not run.
Model
Method
ARC-e
ARC-c
HS
PIQA
WG
LAMB.
Mean
Qwen3-1.7B
BF16
73.4
45.0
66.4
75.5
63.6
63.2
64.5
MXFP4 W4A16 (default)
70.5
43.5
62.9
74.6
63.3
49.0
60.6
MXFP4 W4A16
72.5
44.7
63.9
74.3
62.8
52.3
61.8
W4A8, SC-MXFP4 (ours)
73.2
43.6
63.5
73.6
62.3
55.2
61.9
W4A8, 16-block (ours)
72.7
43.3
64.7
73.8
62.0
58.0
62.4
Qwen3.5-35B-A3B
BF16
85.0
62.4
75.7
84.0
76.3
75.6
76.5
Table 3: Zero-shot accuracy (%) of the deployed systems (Qwen3.5: first 1000 examples per task). MXFP4 and W4A8 quantize the same matrices; “default” is MLX’s conversion, which also quantizes embeddings and head. Normalized accuracy for ARC-c, HellaSwag and PIQA.
Model
Tokens
llama.cpp
MLX MXFP4
W4A8
vs. MLX
vs. llama.cpp
Qwen3-1.7B (dense)
512
41.2
35.4
26.4
1.34×
1.56×
2048
175.0
131.5
75.4
1.74×
2.32×
8192
1020.2
690.5
428.0
1.61×
2.38×
OLMoE-1B-7B (MoE)
512
41.4
38.4
31.9
1.20×
1.30×
2048
158.3
122.5
87.8
1.40×
1.80×
4096
342.7
251.3
167.5
1.50×
2.05×
Table 4: Time to first token (ms) and decode throughput (tokens/s) on an M5 Max. Speedups, from unrounded medians, are W4A8 vs. MLX MXFP4 measured in the same session and vs. llama.cpp (Q4_0, its fastest 4-bit type here), whose prefill was measured in a separate session (the faster of two runs); every prefill ratio against it is at least 1.14× . MoE rows: the default configuration (§ 4 ). Prefill rows use the sustained clock regime (prompts following long ones; Table 9 also gives the burst regime). llama.cpp’s MXFP4_MOE type is 11–26% slower than its Q4_0 on the two MoEs, its Q4_K_M and IQ4_XS prefill at 0.82 – 0.96× Q4_0’s speed (Appendix I ). Decode rows: all three systems back to back in one session per model.
Format
bits/w
W4A16
W4A8
vs. FP8
on M5
NVFP4, E4M3 scale search
4.50
4.904
4.917
+5.5%
FP16
NVFP4, amax/6 rule
4.50
4.918
4.928
+5.7%
FP16
16-block, m≤10 (ours)
4.27
4.970
4.969
+6.6%
INT8, exact
Affine INT4, g64 (MLX 4-bit)
4.50
4.995
4.996
+7.2%
FP16 / INT8 ∗
SC-MXFP4 (ours, std. MXFP4)
4.09
5.097
5.107
+9.6%
INT8, exact
MXFP4, OCP rule (MLX)
4.25
5.199
5.212
+11.8%
FP16
Table 5: Qwen3.8-Flash-Next, WikiText-2 test perplexity (145 windows of 2048 tokens; FP8 checkpoint: 4.662). All methods quantize the same matrices (every projection and expert); NVIDIA’s NVFP4 release quantizes only the routed experts ( NVIDIA, 2026 ) . bits/w: for this model’s matrices (our formats’ row scales add 32/K , and the expert down projections have K=640 ). “vs. FP8”: the W4A8 column, from unrounded perplexities. All perplexities use the model’s reference DeltaNet recurrence. Last column: how the format multiplies on M5 ( ∗ INT8 with a correction per 64-element group, as in oMLX ( oMLX contributors, 2026 ) ).
512
2048
8192
decode
MLX MXFP4 (W4A16)
326
1056
4799
42.4
W4A8, SC-MXFP4 (ours)
254
830
3943
42.4 ∗
W4A8, 16-block (ours)
266
844
3982
27.4
SC-MXFP4 vs. MLX
1.29×
1.27×
1.22×
1.00×∗
16-block vs. MLX
1.23×
1.25×
1.21×
0.65×
Table 6: Qwen3.8-Flash-Next on an M5 Max: time to first token (ms) and batch-1 decode (tokens/s) of our W4A8 kernels with SC-MXFP4 (58.9 GiB) and with the 16-block format (61.6 GiB), and of MLX MXFP4 W4A16 on exactly the same matrices (65.0 GiB). The systems do not fit in memory together, so each was loaded in its own session (two sessions each, alternating; medians agree within 5%). ∗ Decode with MLX’s W4A16 kernels on the same SC-MXFP4 file (two later sessions, 42.4 and 42.5, in which MLX MXFP4 decoded at 42.4); with our W4A8 GEMVs, 37.6 ( 0.89× ).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
M×N×K
FP16
INT8 × INT8
INT8 × INT4
FP16 × INT8
FP16 × INT4
512×4096×4096
54.9
108.7 (1.97)
95.5 (1.74)
45.9 (0.84)
48.9 (0.89)
1024×4096×4096
55.6
110.2 (1.98)
94.9 (1.70)
50.3 (0.90)
49.5 (0.89)
4096×4096×4096
64.4
126.7 (1.97)
116.3 (1.81)
58.5 (0.91)
60.8 (0.94)
2048×8192×8192
59.6
118.9 (1.98)
108.1 (1.81)
53.2 (0.90)
55.8 (0.93)
4096×14336×4096
65.4
127.4 (1.95)
117.8 (1.80)
60.5 (0.92)
62.8 (0.96)
4096×4096×14336
63.9
114.2 (1.79)
111.5 (1.74)
43.5 (0.68)
50.7 (0.79)
Appendix
Table 7: matmul2d throughput (TFLOPS or TOPS, median of 15 interleaved repetitions) on the M5 Max, AC power, High Power mode; speedup vs. FP16 in parentheses (median of per-repetition ratios). One matmul2d call per 128×64 output tile, 4 simdgroups, grouped tile order, weights N×K . All outputs bit-exact.
MLX
matmul2d
MLX
W4A16
W4A16
W4A8
vs MLX MXFP4
M×N×K
FP16
FP16
MXFP4
fused
2-pass
(ours)
W8A8 †
mixed
alike
512 × 4096 × 4096
0.321
0.300
0.346
0.934
0.361
0.220
0.168
1.57 ×
1.48 ×
1024 × 4096 × 4096
0.677
0.583
0.716
1.824
0.666
0.423
0.319
1.69 ×
1.69 ×
2048 × 4096 × 4096
1.360
1.178
1.446
3.443
1.242
0.838
0.628
1.72 ×
1.80 ×
4096 × 4096 × 4096
2.957
2.273
3.212
6.330
2.420
1.581
1.259
2.03 ×
1.86 ×
2048 × 14336 × 4096
4.954
4.000
5.358
11.198
4.348
2.703
1.992
1.98 ×
1.90 ×
Appendix
Table 8: Linear-layer latency (ms, median) on M5 Max; all W4 variants except MLX read the same SC-MXFP4 weights. Our kernels are timed with GPU timestamps: the elapsed time of one command buffer that runs a pipeline’s kernels back to back, divided by the number of runs it contains, so gaps between a pipeline’s kernels count: matmul2d FP16 is a plain FP16 GEMM; W4A16 fused unpacks to FP16 tiles inside the K-loop; W4A16 2-pass unpacks to an FP16 buffer and runs an FP16 GEMM with the row scales; W4A8 quantizes the activations, unpacks to an INT8 buffer and runs an INT8 GEMM, scaling in registers or in a separate pass, whichever is faster; † W8A8 quantizes the activations and runs an INT8 GEMM with INT32 output from 8-bit weights (no unpack, no scaling), a bound on the achievable speed. MLX’s kernels are timed by wall clock in a separate process (10 calls per evaluation), which includes some host overhead, so the “mixed” vs-MLX column mixes two timing methods. “Alike”: our MLX-integrated kernels (the ones used end to end) and MLX’s, called from MLX and timed alike by wall clock in one process, interleaved; at K=14336 the MLX-integrated W4A8 is slower than the standalone one (3.90 vs. 2.70 ms). Like for like are this column and the comparisons against matmul2d FP16 and W4A16 2-pass.
Regime
System
512
2048
4096
8192
Qwen3-1.7B (dense)
burst
MLX MXFP4
31.4
120.4
–
–
W4A8
21.8 (1.44)
70.4 (1.71)
–
–
sustained
MLX MXFP4
35.4
131.5
–
690.5
W4A8
26.4 (1.34)
75.4 (1.74)
–
428.0 (1.61)
OLMoE-1B-7B (MoE, 64 experts)
Appendix
Table 9: Prefill time to first token (ms, median of 7–10) by clock regime, with the speedup over MLX MXFP4 in the same run in parentheses (ratio of medians). MoE: the default configuration (W4A8) and the two expert-weight paths forced at every length. – = not measured.
Qwen3
SmolLM3
OLMoE
Qwen3.5
Method
0.6B
1.7B
4B
3B
1B-7B ‡
35B-A3B ‡
BF16 weights, INT8 activations
12.98
9.64
8.13
8.00
7.59
–
MXFP4, OCP exponent rule, W4A16
15.19
10.81
8.54
8.71
–
–
MXFP4, exponent search, W4A16
15.198
10.697
8.506
8.550
–
–
SC-MXFP4 W4A16
15.195
10.704
8.506
8.548
8.036
6.001
SC-MXFP4 + GPTQ W4A16
–
10.22
–
–
–
–
Appendix
Table 10: WikiText-2 perplexity ablations (same protocol as Table 2 ). The two W4A16 rows with exponent search show three decimals for the comparison in § 5 . – = not run.
bits/w
PPL
Tasks
Prefill TTFT (ms)
Decode
mean
512
2048
8192
tok/s
MLX MXFP4 (W4A16)
4.25
–
61.8
31.5
120.3
654.7
368
NVFP4 (W4A8, fake quant) ∗
4.50
10.51
–
–
–
–
–
SC-MXFP4 (W4A8)
4.08
10.99
61.9
22.0
70.4
395.1
359
16-block, m≤10 (W4A8)
4.26
10.40
62.4
22.3
70.5
390.3
357
Appendix
Table 11: SC-MXFP4 and the 16-wide, integer-multiplier format on the same INT8 kernels (Qwen3-1.7B, M5 Max, one session separate from Table 4 ’s; prefill at 512 and 2048 tokens in the burst regime, at 8192 in the sustained regime). Perplexity: WikiText-2, W4A8, measured on the kernels; NVFP4 is fake-quantized (it has no exact INT8 execution; ∗ with scale search 10.34). Tasks: mean zero-shot accuracy over six lm-evaluation-harness tasks, on the kernels. bits/w: average over this model’s quantized matrices.
Figure 3: Block-exponent spread per weight row. Spread above 3 (shaded) needs an exponent adjustment. Logarithmic axis; its bottom tick marks 0 (no rows).
Model
Matrices
sn≤3
Max sn
MSE, MXFP4 → SC
Qwen3-0.6B
196
99.8%
6
12.70→12.70
Qwen3-1.7B
196
99.7%
6
12.72→12.72
Qwen3-4B
252
97.9%
8
12.74→12.74
SmolLM3-3B
252
99.7%
8
12.65→12.66
OLMoE-1B-7B (MoE)
3136
99.9%
8
12.53→12.53
Qwen3.5-35B-A3B (MoE)
30970
99.97%
7
12.71→12.71
Appendix
Table 12: Rows whose block exponents (MXFP4 with exponent search) span at most 3 over the whole row, and relative weight MSE ( ×103 ) before and after enforcing the constraint, over all transformer-block linear layers. † Every W4A8 matrix of the FP8 checkpoint (95.4M rows; 73,728 of the 74,028 matrices are routed experts), with the FP8 experts decoded exactly.
Qwen3-1.7B
OLMoE-1B-7B
Qwen3.5-35B-A3B
BF16 perplexity, llama.cpp / ours
BF16
8.436 / 8.445
6.724 / 6.727
5.131 / 5.133
llama.cpp: change against its BF16 (bits per weight of the file)
Q4_0
+14.9% (4.90)
+4.3% (4.54)
+3.2% (4.55)
Q4_K_M
+4.8% (5.15)
+2.2% (4.87)
+1.9% (4.89)
IQ4_XS
+8.9% (4.73)
+2.7% (4.34)
+1.7% (4.37)
Appendix
Table 13: llama.cpp’s 4-bit types and ours, scored the same way: llama-perplexity ’s protocol on the WikiText-2 test set (disjoint 2048-token windows, second half of each scored; not comparable with Table 2 ), each against its own BF16. llama.cpp: b11706, Metal. Ours: fake-quantized as in Table 2 (W4A8 is what our kernels compute), about 4.1 (SC-MXFP4) and 4.3 (16-block) bits per weight in the quantized matrices, embedding and output layer in BF16. Speed: relative to Q4_0 in the same session, prefill at 512, 2048 and 8192 tokens (OLMoE: 4096; the faster of two batch sizes) and decode at a 512-token context; MXFP4_MOE’s speed is in Table 4 . – = not run or not applicable.
Quantization is essential for efficient large language model (LLM) inference, yet the dequantization step-converting low-bit weights back to high-precision for matrix multiplication has become a critical bottleneck on modern AI accelerators. On architectures with decoupled compute units (e.g., Ascend NPUs), dequantization operations can consume more cycles than the matrix multiplication itself, leaving the high-throughput tensor cores underutilized. This paper presents Multi-Scale Dequant (MSD), a quantization framework that removes weight/KV dequantization from the GEMM critical path. Instead of lifting low-bit weights to BF16 precision, MSD decomposes high-precision BF16 activations into multiple low-precision components, each of which can be multiplied directly with quantized weights via native hardware-accelerated GEMM. This approach shifts the computational paradigm from precision conversion to multi-scale approximation, avoiding INT8-to-BF16 weight conversion before GEMM. We instantiate MSD for two weight formats and derive tight error bounds for each. For INT8 weights (W4A16), two-pass INT8 decomposition achieves near 16 effective bits. For MXFP4 weights (W4A16), two-pass MXFP4 decomposition yields near 6.6 effective bits with error bound 1/64 per block surpassing single-pass MXFP8(5.24 bits) while maintaining the same effective GEMM compute time. We further derive closed-form latency and HBM traffic models showing that MSD avoids the Vector-Cube pipeline stall caused by dequantization and reduces KV cache HBM traffic by up to 2.5 times in attention. Numerical simulations on matrix multiplication and Flash Attention kernels confirm that MSD does not degrade accuracy compared to dequantization baselines, and in many settings achieves lower L2 error.
Training large language models (LLMs) in sub-8-bit arithmetic promises substantial gains in memory bandwidth and compute throughput, yet FP4 formats introduce quantization error severe enough to destabilize optimization if left unaddressed. We present ALTO, an open-source training recipe for GPT-OSS-20B, a 20-billion-parameter mixture-of-experts (MoE) model, on the MLPerf Small MoE Training benchmark under the MXFP4 microscaling format. The recipe combines established techniques from Abecassis, Felix et al. [arXiv:2509.25149] (hybrid 1D/2D block quantization, randomized Hadamard transforms, and stochastic rounding) with a simplified weight de-oscillation scheme adapted from TetraJet-v2 [arXiv:2510.27527]. On C4 validation at global batch size 16, MXFP4 + RHT + SR + de-oscillation reaches a validation loss of 3.3350 at 16,128 steps, only 0.007 above the BF16 baseline (3.3283), closing roughly 50% of the gap left by MXFP4 without de-oscillation (3.3418). Measured by steps to reach the MLPerf quality target (validation loss 3.34), de-oscillation cuts the FP4 convergence overhead relative to BF16 from +20.0% to +5.0%. We also report negative end-to-end results from differential gradient estimation, outlier clipping, and macro-block scaling, despite each showing operator-level SNR gains. All experiments use simulated MXFP4 kernels on AMD MI300X hardware and wall-clock speed has not been optimized.
Four-bit floating-point (FP4) Tensor Cores accelerate matrix multiplication, but scale computation, operand packing, layout construction, and saved backward state can erase the gain. We present format-aware fusion, which co-designs each quantization producer with its scale domain and consumer layout for native \mxfp{}, global \nvfp{}, and cooperative-thread-array-local \nvfp{}. We evaluate Llama-3-family 8B pretraining through 160 billion tokens using bfloat16 output projections and compiled cross entropy. In matched same-accelerator probes, bfloat16 and Transformer Engine \nvfp{} reach 18.8K and 27.6K tokens/s/GPU, while our fastest custom route reaches 37.9K. \mxfp{} with row-gradient stochastic rounding and fixed-sign 32-value Hadamard weight-gradient preconditioning reaches 37.2K tokens/s/GPU (86.3% bfloat16 model FLOP utilization) and ends 2.11% above the raw bfloat16 training-loss endpoint. A Transformer Engine recipe with four final bfloat16 blocks ends 0.87% above bfloat16 at 27.1K tokens/s/GPU. Downstream rankings differ from training-loss rankings, showing that FP4 outcomes depend jointly on scale contract, operand, and execution path.