Recommendation and Ranking (R&R) models, particularly in digital advertising, require massive inference throughput but exhibit extreme sensitivity to precision loss. Unlike Large Language Models (LLMs) that frequently tolerate quantization-induced accuracy drops of 1% or more, production-grade R&R models enforce a stringent launch bar: model quality must remain within a 0.01% Normalized Entropy (NE) margin relative to the baseline, which commonly uses FP16 or FP32. This tight constraint has historically hindered the adoption of aggressive quantization in this domain. In this paper, we present an empirical evaluation of both MXFP8 and the 6-bit MXFP6 format for industrial-scale R&R inference on the AMD MI350X architecture. To the best of our knowledge, this is the first paper, across R&R and LLM domains, that reports significant speedups from native MXFP6 GEMMs, which quantize both weights and activations and compute directly on 6-bit elements, without sacrificing model quality. We demonstrate that we can selectively apply this 6-bit format to maximize overall QPS gain without breaching the strict NE parity threshold. We further show that MXFP8 is a viable alternative to row-wise FP8: the two formats have nearly the same accuracy on our models, so the faster kernel can be chosen for each GEMM shape. On two production ads ranking models, our implementation shows measured GEMM kernel speedups over FP16 of 1.33--1.55x for MXFP8 and 1.73--1.99x for MXFP6 across all supported linear layers, and up to 2.90x for MXFP6 on individual layers, alongside end-to-end (E2E) inference throughput gains from selective MXFP6 of 1.5--5.3% for Model A and 7.6--14.3% for Model B over their FP8 configurations.
Figures & tables
Precision
NVIDIA GB300
AMD MI350X
FP16
1×
1×
FP8 / MXFP8
2×
2×
MXFP6
2×
4×
MXFP4
6×
4×
NVFP4
6×
–
Table 1: Dense peak tensor throughput of each precision, relative to FP16 NVIDIA (2024) ; Advanced Micro Devices, Inc. (2026a) . MI350X has twice the MXFP8 peak throughput for MXFP6.
Figure 1: Inference work per example of the most complex launched late-stage ads ranking model at Meta, normalized to H1’24 ( 1× ), on a logarithmic scale. The sharp increase in H2’25 coincides with the first launch of the Adaptive Ranking Model Meta (2026b) ; H2’26 is an estimated target.
Format
Exp.
Man.
Fmax
Fmin
Dyn. Range
Scale
Granularity
Bits
FP16
5
10
65,504
2−24
≈1.10×1012
–
–
16
FP8 (tensor)
4
3
448
2−9
229,376
FP32
Whole tensor
8
FP8 (row)
4
3
448
2−9
229,376
FP32
Row (act.), channel (wt.)
≈8
MXFP8
4
3
448
2−9
229,376
E8M0
Block of 32
8.25
MXFP6
2
3
7.5
2−3
60
E8M0
Block of 32
6.25
NVFP4
2
1
6
2−1
12
E4M3 + FP32
Block of 16 + tensor
4.5
Table 2: Low-precision formats discussed in this paper. All formats have one sign bit. Bits per element include the amortized storage of the scales; FP16 is the baseline. MXFP8 and MXFP6 are listed with their high-mantissa variants (E4M3 and E2M3, respectively).
QSNR (dB)
Speedup over FP16
Speedup over FP16
Batch 1024
Batch 2048
Format
Min
P10
Median
Mean
Total
Per-shape
Shapes <1×
Total
Per-shape
Shapes <1×
Model A, 20 most compute-intensive linear layers (6 shapes)
FP16
70.9
71.1
71.7
71.8
1.00×
1.00×
–
1.00×
1.00×
–
FP8 (row)
32.5
33.1
34.6
35.5
1.64×
1.68×
0
1.82×
1.73×
0
MXFP8
32.3
32.8
34.6
35.4
1.67×
1.71×
0
1.70×
1.68×
0
Table 3: QSNR distribution and GEMM speedup over FP16 for the 20 most compute-intensive linear layers and for all linear layers of the two production models. For accuracy only, the FP16 row reports QSNR against FP32; the low-precision rows report QSNR against FP16. All speedups use FP16 execution time as the baseline, so FP16 is 1.00× by definition. MXFP6 speedups are those of the deep-pipelined kernel. Total speedup is the ratio of the summed GEMM time; per-shape speedup is the geometric mean over unique shapes. The all-layer blocks cover the call sites that every format supports.
Component
Label
Layers
K
N
Model A
MLP, wide input
MLP-W
1
1.6e5
6.1e3
MLP
MLP
7
2.3e4
6.1e3
Projection
Proj.
9
6.1e3
3.3e4
DCN down-proj.
DCN-D
1
1.6e5
1.0e3
DCN up-proj.
DCN-U
1
1.0e3
1.6e5
Table 4: Linear shapes of the 20 most compute-intensive layers of each model, grouped by component. K is the input dimension and N the output dimension; ranges indicate multiple shapes. Dimensions are rounded to two significant digits; aeb denotes a×10b .
Model
Parameters quantized to MXFP6
E2E QPS improvement
Tasks within 0.01% NE tolerance
Model A
80.8%
+5.3% / +1.5%
6 / 6
Model B
72.9%
+14.3% / +7.6%
3 / 3
Table 5: Summary of MXFP6 accuracy and end-to-end performance. Percentages use all unique linear-layer weights as the denominator, excluding biases and embeddings. QPS improvements are over the production FP8 configurations at batches 1024 / 2048, respectively.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Batch 1024
Batch 2048
Work band
Shapes
FP16 time
FP8
MXFP8
MXFP6
MXFP6 vs. MXFP8
FP16 time
FP8
MXFP8
MXFP6
MXFP6 vs. MXFP8
Model A
Smallest
11
4.4%
1.05
1.13
1.17
1.03
2.9%
1.07
1.18
1.20
1.02
Small
16
8.0%
1.23
1.00
1.17
1.16
6.5%
1.32
1.17
1.28
1.09
Medium
12
34.1%
1.46
1.31
1.65
1.26
33.3%
1.50
1.50
1.86
1.23
Largest
6
53.5%
1.62
1.66
2.29
1.38
57.2%
1.81
1.67
2.29
1.37
Appendix
Table 6: GEMM speedup by relative GEMM size, for the call sites that every format supports. FP16 time is the share of the total FP16 GEMM time; the FP8, MXFP8, and MXFP6 columns give the speedup over FP16, and the last column of each batch size gives the speedup of MXFP6 over MXFP8. Work per call is normalized to the largest supported shape of each model at batch 1024. The four bands have boundaries at approximately 0.05%, 0.50%, and 4.98% of that reference for Model A, and 0.12%, 1.18%, and 11.76% for Model B. Shapes keep the same band at batch 2048.
QSNR (dB)
Speedup, batch 1024
Speedup, batch 2048
Component
Label
Layers
K
N
FP8
MXFP8
MXFP6
FP8
MXFP8
MXFP6
FP8
MXFP8
MXFP6
Model A
MLP, wide input
MLP-W
1
1.6e5
6.1e3
32.9
32.8
31.2
1.82
1.96
2.72
2.19
1.94
2.78
MLP
MLP
7
2.3e4
6.1e3
34.8
34.7
33.1
1.40
1.55
2.46
1.63
1.32
2.27
Projection
Proj.
9
6.1e3
3.3e4
34.0
33.9
32.0
1.70
1.58
2.05
1.83
1.83
2.19
DCN down-proj.
DCN-D
1
1.6e5
1.0e3
40.8
40.8
39.8
2.52
2.11
2.85
1.98
2.06
2.90
Appendix
Table 7: QSNR and GEMM speedup over FP16 for the 20 most compute-intensive linear layers of each model, grouped by model component. QSNR is the mean over the layers of each component, and speedup is the ratio of their summed GEMM time. In Model A, all layers of a component share one GEMM shape; in Model B, some components span several shapes. Linear dimensions are rounded to two significant digits; aeb denotes a×10b .
Figure 2: Accuracy and performance of the 20 most compute-intensive linear layers of each model, grouped by model component (labels as in Table 4 ). Speedup bars start at the FP16 baseline ( 1× ).
Quantization-aware training (QAT) is typically performed for a single target numeric format, while practical deployments often need to choose numerical precision at inference time based on hardware support or runtime constraints. We study multi-format QAT, where a single model is trained to be robust across multiple quantization formats. We find that multi-format QAT can match single-format QAT at each target precision, yielding one model that performs well overall across different formats, even formats that were not seen during training. To enable practical deployment, we propose the Slice-and-Scale conversion procedure for both MXINT and MXFP that converts a high-precision representation into lower-precision formats without re-training. Building on this, we introduce a pipeline that (i) trains a model with multi-format QAT, (ii) stores a single anchor format checkpoint (MXINT8/MXFP8), and (iii) allows on-the-fly conversion to lower MXINT or MXFP formats at runtime with negligible-or no-additional accuracy degradation. Together, these components provide a practical path to elastic precision scaling and allow selecting the runtime format at inference time across diverse deployment targets.
On hardware whose fastest matrix datapath is INT8×INT8, such as the Neural Accelerators of Apple's M5 GPUs, 4-bit weights must become INT8 operands. Existing MXFP4 kernels already map FP4 (E2M1) values to integers by doubling them, but apply every 32-element block scale inside the reduction. We show when no per-block work is needed: a row of MXFP4 weights is exactly one INT8 vector times one power-of-two scale if its block exponents span at most three, so a linear layer becomes a single INT8 GEMM with one rescale per output. Enforcing the condition adds at most 1/16 of a row's largest rounding error to the blocks it changes, and real weights already satisfy it: 97.9-99.97% of rows across seven models, including 99.93% of the 95M rows of a 125B-parameter mixture-of-experts (MoE) model, Qwen3.8-Flash-Next. The resulting spread-constrained MXFP4 files stay standard MXFP4. With integer instead of power-of-two block multipliers, an integer-exact format (16-wide blocks, multipliers 1..10, 4.26 bits) runs as one exact INT8 GEMM and is more accurate than NVFP4 (4.5 bits) with its standard scale rule on five small models, within 0.06 perplexity of NVFP4 with scale search. On Qwen3.8-Flash-Next's routed experts with INT8 activations it is as close to the FP8 model as NVFP4 with scale search (KL divergence 0.0557 vs. 0.0553) and closer than MLX's MXFP4 and affine INT4; with every matrix quantized it solves 94.8% of GSM8K vs. 91.0% for MLX's MXFP4. Quantizing only the 74k routed expert matrices (96% of the parameters) to either of our formats with INT8 activations costs at most 0.2% perplexity. A complete W4A8 system on the M5 produces outputs bit-identical to the quantized reference; prefill is 1.34-1.74x faster than MLX on a dense model, 1.10-1.57x on two MoE models up to 35B parameters and 1.22-1.29x on Qwen3.8-Flash-Next, where shared non-GEMM work bounds the gain.
Shinyoung Kang, Junhyung Ahn, Yujun Kim +1
UT Southwestern Medical Center · Department of Data and Decision Sciences, Emory University · Fuqua School of Business, Duke University +1
As the demand for deep learning grows, cost reduction through quantization has become essential for both training and inference. In 2022, the Open Compute Project (OCP) consortium standardized narrow precision formats for deep learning, called the microscaling (MX) format. The MX format is a hardware-friendly dynamic quantization scheme that effectively reduces the data size by sharing an 8-bit exponent across multiple operands. The MX format can be categorized into two types with their own strengths: (i) MXINT which focuses on a high precision consisting only of mantissa bits and (ii) MXFP which focuses on a wider dynamic range by allowing local exponent bits. In this work, we present a versatile MXFP format, called MX-SAFE (MXSF in short), that adaptively uses two modes, i.e., a wider mantissa mode (FP8 E2M5) and a subnormal FP mode (FP5 E3M2), to support both training and direct-cast inference. Furthermore, we propose a tile-based block design to increase hardware efficiency by reducing the burden of re-quantization process during the training with the MXSF format. Owing to the use of the proposed MXSF format, 0.05%/11.1% and 3.55%/3.57% improvements in accuracy, on average, for inference/full-training compared to MXFP8 E2M5 and MXFP8 E4M3 are observed, respectively. Moreover, we present a training-inference accelerator that supports the MXSF format and it achieves similar accuracy to the BF16 baseline while using 24.9% less total energy consumption.
Dahoon Park, Jahyun Koo, Sangwoo Hwang +1
† School of Electrical Engineering, Korea University, South Korea · ‡ Department of Electrical Engineering and Computer Science, DGIST, South Korea