MF-QAT: Multi-Format Quantization-Aware Training for Elastic Inference

Zifei Xu; Sayeh Sharify; Hesham Mostafa

cs.LGApr 1, 2026

MF-QAT: Multi-Format Quantization-Aware Training for Elastic Inference

Authors: Zifei Xu, Sayeh Sharify, Hesham Mostafa

Organizations: d-Matrix, Santa Clara, CA 95054

Abstract

Quantization-aware training (QAT) is typically performed for a single target numeric format, while practical deployments often need to choose numerical precision at inference time based on hardware support or runtime constraints. We study multi-format QAT, where a single model is trained to be robust across multiple quantization formats. We find that multi-format QAT can match single-format QAT at each target precision, yielding one model that performs well overall across different formats, even formats that were not seen during training. To enable practical deployment, we propose the Slice-and-Scale conversion procedure for both MXINT and MXFP that converts a high-precision representation into lower-precision formats without re-training. Building on this, we introduce a pipeline that (i) trains a model with multi-format QAT, (ii) stores a single anchor format checkpoint (MXINT8/MXFP8), and (iii) allows on-the-fly conversion to lower MXINT or MXFP formats at runtime with negligible-or no-additional accuracy degradation. Together, these components provide a practical path to elastic precision scaling and allow selecting the runtime format at inference time across diverse deployment targets.

Figures & tables

Panel of Figure 1 — Figure 1: Multi-format QAT on Qwen3-4B-Base , trained for one epoch on 128 WikiText-2 examples. Left: MXINT formats. Right: MXFP formats. For each training variant, we show the result at the best learning rate, chosen as the one with the lowest evaluation loss among the three learning rates in Section 3.2 . The x-axis denotes evaluation quantization bit-width, and the y-axis denotes WikiText-2 validation perplexity after post-training quantization to the corresponding bit-width. The horizontal dashed line marks the original high-precision model (FP16/BF16/FP32, depending on pretraining). The black solid line corresponds to full-precision fine-tuning, the purple solid line to multi-format QAT, and the other colored solid lines to single-format QAT.

Model	QAT/FT Precision	PTQ Precision
Model	QAT/FT Precision	MXINT2	MXINT3 ∗	MXINT4	MXINT5 ∗	MXINT6	MXINT7 ∗	MXINT8
	Full Precision FT	22.7	37.5	44.4	44.7	45.8	46.0	46.0
	MXINT2 QAT	29.3	39.6	44.4	44.6	45.1	45.6	45.1
	MXINT4 QAT	22.9	38.7	44.6	44.8	45.6	45.9	46.0
	MXINT6 QAT	22.1	38.4	43.7	44.6	45.2	46.1	46.0
	MXINT8 QAT	21.9	37.7	44.2	44.8	45.2	46.0	46.0

Table 1: Average 0-shot accuracy (higher is better) on MMLU ( Hendrycks et al., 2021 ) , MathQA ( Amini et al., 2019 ) , and HellaSwag ( Zellers et al., 2019 ) for the MXINT formats . Each row corresponds to a different QAT/FT training precision, and each column shows the PTQ evaluation precision. Columns marked with ∗ denote precisions not seen during multi-format/QAT training. QAT, PTQ, and FT denote Quantization Aware Training, Post Training Quantization, and Finetuning, respectively. Detailed per-individual task results in Appendix B , Tables 4 - 6 .

Model	QAT/FT Precision	PTQ Precision
Model	QAT/FT Precision	MXFP4	MXFP5 ∗	MXFP6	MXFP7 ∗	MXFP8
	Full Precision FT	43.3	45.5	45.7	46.0	46.2
	MXFP4 QAT	43.7	46.0	46.1	45.9	45.6
	MXFP6 QAT	43.3	45.5	45.8	45.7	45.8
	MXFP8 QAT	43.6	45.4	45.7	46.0	45.8
Llama-2-7B	Multi-format QAT	43.5	45.5	45.8	45.8	45.8

Table 2: Average 0-shot accuracy (higher is better) on MMLU ( Hendrycks et al., 2021 ) , MathQA ( Amini et al., 2019 ) , and HellaSwag ( Zellers et al., 2019 ) for the MXFP formats . Each row corresponds to a different QAT/FT training precision, and each column shows the PTQ evaluation precision. Columns marked with ∗ denote precisions not seen during multi-format/QAT training. QAT, PTQ, and FT denote Quantization Aware Training, Post Training Quantization, and Finetuning, respectively. Detailed per-individual task results in Appendix B , Table 7 .

Model	QAT/FT Precision	PTQ Precision
Model	QAT/FT Precision	MXINT4	MXINT5 ∗	MXINT6	MXINT7 ∗	MXINT8
	Full Precision FT	15.0	52.0	36.0	44.0	35.0
	MXINT4 QAT	23.0	55.0	32.0	39.0	38.0
	MXINT6 QAT	17.0	50.0	37.0	41.0	40.0
	MXINT8 QAT	20.0	47.0	40.0	40.0	32.0
Qwen3-VL-2B	Multi-format QAT	15.0	48.0	31.0	34.0	34.0

Table 3: ChartQA ( Masry et al., 2022 ) accuracy for Qwen3-VL ( Bai et al., 2025 ) multi-modal models when quantized to the MXINT and MXFP formats with different precisions. Rows indicate the QAT/FT training precision; columns indicate the PTQ inference precision. Columns marked with ∗ denote precisions not seen during multi-format/QAT training. QAT, PTQ, and FT denote Quantization Aware Training, Post Training Quantization, and Finetuning, respectively.

Panel of Figure 2 — Figure 2: WikiText-2 perplexity for Llama-3.2-1B , comparing direct MXINT quantization and SSMXINT. Left: Varying bit precision at block size 64. Right: Varying block size at 4-bit precision. Horizontal dashed line denotes the baseline model.

Panel of Figure 3 — Figure 3: WikiText-2 perplexity for Llama-3.2-1B , comparing direct MXFP quantization and SSMXFP. Left: Varying bit precision at block size 64. Right: Varying block size at 4-bit precision. Horizontal dashed line denotes the baseline model.

Panel of Figure 4 — Figure 4: Multi-format QAT with Slice-and-Scale on Qwen3-4B-Base . Left: MXINT formats. Right: MXFP formats. Plotting conventions match Figure 1 .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Multi-format QAT results for Llama-3.2-1B .

Panel of Figure 6 — Figure 6: Multi-format QAT results for Llama-3.2-3B .

Panel of Figure 7 — Figure 7: Multi-format QAT results for Qwen3-0.6B-Base .

Panel of Figure 8 — Figure 8: Multi-format QAT results for Qwen3-1.7B-Base .

Panel of Figure 9 — Figure 9: Multi-format QAT results for Qwen3-VL-2B-Instruct .

Panel of Figure 10 — Figure 10: Multi-format QAT results for Qwen3-VL-4B-Instruct .

Panel of Figure 11 — Figure 11: Multi-format QAT results for llama-2-7b .

Panel of Figure 12 — Figure 12: Multi-format QAT with Slice-and-Scale results for Llama-3.2-1B .

Panel of Figure 13 — Figure 13: Multi-format QAT with Slice-and-Scale results for Llama-3.2-3B .

Panel of Figure 14 — Figure 14: Multi-format QAT with Slice-and-Scale results for Qwen3-0.6B-Base .

Panel of Figure 15 — Figure 15: Multi-format QAT with Slice-and-Scale results for Qwen3-1.7B-Base .

Panel of Figure 16 — Figure 16: Multi-format QAT with Slice-and-Scale results for Qwen3-VL-2B-Instruct .

Panel of Figure 17 — Figure 17: Multi-format QAT with Slice-and-Scale results for Qwen3-VL-4B-Instruct .

Panel of Figure 18 — Figure 18: Multi-format QAT with Slice-and-Scale results for llama-2-7b .

Model	QAT/FT Precision	PTQ Precision
Model	QAT/FT Precision	MXINT2	MXINT3 ∗	MXINT4	MXINT5 ∗	MXINT6	MXINT7 ∗	MXINT8
	Full Precision FT	25.0	28.1	39.3	41.1	42.3	42.5	42.7
	MXINT2 QAT	24.5	29.4	39.2	40.1	41.2	41.7	41.0
	MXINT4 QAT	25.3	28.6	39.7	41.8	42.7	42.6	42.9
	MXINT6 QAT	24.5	28.0	38.5	41.1	41.9	42.6	42.5
	MXINT8 QAT	24.3	27.1	39.1	40.7	42.1	42.4	42.7

Appendix

Table 4: MMLU accuracy ( Hendrycks et al., 2021 ) for various models when quantized to the MXINT formats. Each row corresponds to a different QAT/FT training precision, and each column shows the PTQ evaluation precision. Columns marked with ∗ denote precisions not seen during multi-format/QAT training. QAT, PTQ, and FT denote Quantization Aware Training, Post Training Quantization, and Finetuning, respectively.

Model	QAT/FT Precision	PTQ Precision
Model	QAT/FT Precision	MXINT2	MXINT3 ∗	MXINT4	MXINT5 ∗	MXINT6	MXINT7 ∗	MXINT8
	Full Precision FT	18.4	25.0	27.8	28.0	28.8	28.4	28.4
	MXINT2 QAT	20.6	26.4	28.0	29.0	28.2	28.4	27.8
	MXINT4 QAT	17.8	26.4	27.6	27.6	27.8	28.6	28.2
	MXINT6 QAT	15.0	26.8	26.8	27.4	27.2	28.4	28.0
	MXINT8 QAT	16.6	25.6	27.4	28.0	27.4	28.6	28.0

Appendix

Table 5: MathQA normalized accuracy ( Amini et al., 2019 ) for various models when quantized to the MXINT formats. Each row corresponds to a different QAT/FT training precision, and each column shows the PTQ evaluation precision. Columns marked with ∗ denote precisions not seen during multi-format/QAT training. QAT, PTQ, and FT denote Quantization Aware Training, Post Training Quantization, and Finetuning, respectively.

Model	QAT/FT Precision	PTQ Precision
Model	QAT/FT Precision	MXINT2	MXINT3 ∗	MXINT4	MXINT5 ∗	MXINT6	MXINT7 ∗	MXINT8
	Full Precision FT	24.8	59.4	66.2	65.0	66.2	67.2	66.8
	MXINT2 QAT	42.8	63.0	66.0	64.8	65.8	66.6	66.6
	MXINT4 QAT	25.6	61.2	66.4	65.0	66.2	66.6	67.0
	MXINT6 QAT	26.8	60.4	65.8	65.2	66.4	67.4	67.4
	MXINT8 QAT	24.8	60.4	66.2	65.6	66.0	67.0	67.2

Appendix

Table 6: HellaSwag normalized accuracy ( Zellers et al., 2019 ) for various models when quantized to the MXINT formats. Each row corresponds to a different QAT/FT training precision, and each column shows the PTQ evaluation precision. Columns marked with ∗ denote precisions not seen during multi-format/QAT training. QAT, PTQ, and FT denote Quantization Aware Training, Post Training Quantization, and Finetuning, respectively.

Llama-2-7B
QAT/FT Precision	PTQ Precision
	MXFP4			MXFP5 ∗			MXFP6			MXFP7 ∗			MXFP8
	MMLU	Math	HellaS	MMLU	Math	HellaS	MMLU	Math	HellaS	MMLU	Math	HellaS	MMLU	Math	HellaS
Full Prec. FT	37.4	28.8	63.6	42.0	28.6	65.8	42.4	28.4	66.4	42.3	28.4	67.4	42.3	28.8	67.4
MXFP4 QAT	37.8	29.6	63.8	42.5	28.4	67.2	42.5	28.2	67.6	42.6	28.2	67.0	42.1	27.6	67.0
MXFP6 QAT	36.7	29.0	64.2	41.9	28.6	66.0	42.5	28.4	66.6	41.8	28.4	67.0	41.9	28.4	67.2

Appendix

Table 7: Comparison of MMLU ( Hendrycks et al., 2021 ) , MathQA ( Amini et al., 2019 ) , and HellaSwag ( Zellers et al., 2019 ) accuracies for various models when quantized to the MXFP formats. Each row corresponds to a different QAT/FT training precision, and each column shows the PTQ evaluation precision. Columns marked with ∗ denote precisions not seen during multi-format/QAT training. QAT, PTQ, and FT denote Quantization Aware Training, Post Training Quantization, and Finetuning, respectively.

Panel of Figure 19 — Figure 19: Tensor reconstruction MSE for direct MXINT quantization and Slice-and-Scale conversion (SSMXINT) on 100 random tensors. Left: Varying bit precision at block size 64. Right: Varying block size at 4-bit precision.

Panel of Figure 20 — Figure 20: Tensor reconstruction MSE for direct MXFP quantization and Slice-and-Scale conversion (SSMXFP) on 100 random tensors. Left: Varying bit precision at block size 64. Right: Varying block size at 4-bit precision.

Explore similar work

MX-SAFE: Versatile Inference- and Training-Proof Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocation
May 23, 2026Dahoon Park, Jahyun Koo, Sangwoo Hwang +1Mixed-Precision Quantization AI Accelerator Inference
† School of Electrical Engineering, Korea University, South Korea · ‡ Department of Electrical Engineering and Computer Science, DGIST, South Korea
VisionMX: Unlocking Microscaling Post-Training Quantization for Vision Models
Oct 2, 2026Elad Dror Cohen, Ofir Gordon, Lior Dikstein +2Neural Network Quantization Post-Training Quantization
Arm AI Research (AAIR), Arm Ltd
Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
Oct 6, 2026Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2LLM Quantization Quantization-Aware Training
ISTA · EPFL, ISTA

May 23, 2026J/K move · Enter open · S save

MX-SAFE: Versatile Inference- and Training-Proof Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocation

Dahoon Park, Jahyun Koo, Sangwoo Hwang, Jaeha Kung

† School of Electrical Engineering, Korea University, South Korea · ‡ Department of Electrical Engineering and Computer Science, DGIST, South Korea

Mixed-Precision Quantization AI Accelerator Inference Energy-Efficient ML Neural Network Quantization

Open paper AskPDF arXiv

As the demand for deep learning grows, cost reduction through quantization has become essential for both training and inference. In 2022, the Open Compute Project (OCP) consortium standardized narrow precision formats for deep learning, called the microscaling (MX) format. The MX format is a hardware-friendly dynamic quantization scheme that effectively reduces the data size by sharing an 8-bit exponent across multiple operands. The MX format can be categorized into two types with their own strengths: (i) MXINT which focuses on a high precision consisting only of mantissa bits and (ii) MXFP which focuses on a wider dynamic range by allowing local exponent bits. In this work, we present a versatile MXFP format, called MX-SAFE (MXSF in short), that adaptively uses two modes, i.e., a wider mantissa mode (FP8 E2M5) and a subnormal FP mode (FP5 E3M2), to support both training and direct-cast inference. Furthermore, we propose a tile-based block design to increase hardware efficiency by reducing the burden of re-quantization process during the training with the MXSF format. Owing to the use of the proposed MXSF format, 0.05%/11.1% and 3.55%/3.57% improvements in accuracy, on average, for inference/full-training compared to MXFP8 E2M5 and MXFP8 E4M3 are observed, respectively. Moreover, we present a training-inference accelerator that supports the MXSF format and it achieves similar accuracy to the BF16 baseline while using 24.9% less total energy consumption.

Abstract

Explore similar work

MX-SAFE: Versatile Inference- and Training-Proof Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocation

VisionMX: Unlocking Microscaling Post-Training Quantization for Vision Models

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training