MXCode: Vector Quantization for MX-Native Inference at Scale

Gaurav Kolhe; Cansu Demirkiran; Shubhi Tyagi; Sulaiman Sadiq; Boyao Wang; Richard Wang; Maximiliana Behnke; Shreyas Labhsetwar

cs.LGOct 30, 2026

MXCode: Vector Quantization for MX-Native Inference at Scale

Authors: Gaurav Kolhe, Cansu Demirkiran, Shubhi Tyagi, Sulaiman Sadiq, Boyao Wang, Richard Wang, Maximiliana Behnke, Shreyas Labhsetwar,

Organizations: Microsoft · Carnegie Mellon University

Abstract

Memory capacity and bandwidth are major bottlenecks in LLM inference, motivating weight compression beyond conventional 4-bit quantization. Vector quantization (VQ) enables approximately 2-bit weight storage but often relies on codebooks incompatible with native FP4 computation. We introduce MXCode (MX-Native Codebook Quantization), a framework bridging ultra-low-bit VQ compression with MicroXcaling (MX)-native FP4 computation. MXCode proposes three strategies: PostScale, which quantizes learned codebooks to MXFP4; PreScale, which extracts MX scales before codebook learning; and PreScale+, which further compresses these scales using VQ. These strategies explore tradeoffs among quantization quality, scale overhead, and codebook granularity. All reconstruct MX-native W4A4 operands without materializing intermediate high-precision weights. We evaluate MXCode on Qwen3 (235B), DeepSeek-R1 (671B), and GLM-5.2 (753B) using the Microsoft MX Emulation Library. MXCode achieves approximately 2 bits per weight with competitive, workload-dependent model quality. On GB300 hardware, experiments with Qwen3 (235B) show approximately 50% lower model-load memory than Native FP4 while maintaining near-Native-FP4 serving throughput. Under memory-constrained prefix reuse, P90 TTFT improves by up to 94.5%. These results demonstrate software-based reconstruction on existing FP4 hardware and inform the development of future MX-native inference systems.

Figures & tables

Format	Block size	Scale	Min. positive scale	Max. finite scale	Optional tensor scale	BPE
E8M0-B32-FP4	32	E8M0	$2^{-127}$	$1.70\times 10^{38}$	–	4.25
E4M3-B16-FP4	16	E4M3	$2^{-9}$	$448$	FP32	4.50
E5M3-B16-FP4	16	UE5M3	$2^{-17}$	$114688$	FP32	4.50

Table 1: Block-scaled FP4 formats compared in this work. All use E2M1 elements and 8-bit block scales. The reported ranges refer to block scales, and BPE excludes tensor-level scaling overhead.

Figure 1: MXCode: MX-native codebook quantization pipeline; (a) PostScale, (b) PreScale, and (c) PreScale+ compression schemes.

Figure 2: Aggregate weight SQNR as a function of effective codebook size for different VQ dimensions

d

. Each point corresponds to a different effective BPE and number of centroids

K

, indicated by its numeric label. BPE calculation only includes VQ indices. The shaded upper-left region denotes configurations that satisfy both design constraints: SQNR

\geq 10

dB and LUT size

\leq 1

KB.

Figure 3: Ablation of FP4 codebook quantization. Each color shows the QBlock-size sweep from full tensor to

64\times 64

for a VQ/MXCode scheme and datatype. The BF16-centroid baseline is independent of the MX scale format and is shown in gray across all three plots. From left to right, charts shows (a) QBlock-size sweep for E4M3-B16-FP4 and E8M0-B32-FP4. (b) QBlock-size sweep at various scale compression ratios for E4M3-B16-FP4 MXCode-PreScale+ . (c) QBlock-size sweep at various scale compression ratios for E8M0-B32-FP4 MXCode-PreScale+ . Effective storage includes the VQ indices, FP4 codebooks, MX scales, and scale-VQ metadata.

	Compute format	E4M3-B16-FP4				E8M0-B32-FP4
	Method	Native FP4	MXCode PostScale	MXCode PreScale	MXCode PreScale+	Native FP4	MXCode PostScale	MXCode PreScale	MXCode PreScale+
	Weight (BPE)	4.50	2.06	2.56	2.13	4.25	2.25	2.50	2.28
Qwen3 (235B)	MMLU-Pro	82.41 $\pm 0.34$	79.18 $\pm 0.36$	80.34 $\pm 0.35$	80.54 $\pm 0.35$	82.02 $\pm 0.34$	79.67 $\pm 0.35$	79.09 $\pm 0.36$	79.00 $\pm 0.36$
	RULER	95.05 $\pm 0.14$	94.16 $\pm 0.15$	95.01 $\pm 0.14$	95.00 $\pm 0.14$	95.13 $\pm 0.13$	94.69 $\pm 0.14$	94.73 $\pm 0.14$	94.83 $\pm 0.14$
	DROP	88.56 $\pm 0.33$	86.98 $\pm 0.34$	86.54 $\pm 0.35$	87.69 $\pm 0.34$	87.96 $\pm 0.33$	85.55 $\pm 0.36$	87.52 $\pm 0.34$	86.35 $\pm 0.35$
	GSM8K	96.89 $\pm 0.48$	96.66 $\pm 0.49$	97.12 $\pm 0.46$	96.36 $\pm 0.52$	96.89 $\pm 0.48$	96.82 $\pm 0.48$	96.44 $\pm 0.51$	96.21 $\pm 0.53$

Table 2: W4A4 evaluation across three models. Accuracy and standard errors are reported as percentages. Native FP4 uses W4A4 computation with either the E4M3-B16-FP4 or E8M0-B32-FP4 format; without any VQ or MXCode. Weight storage, measured in BPE, includes VQ indices, FP4 codebooks, and scales, as applicable. Average accuracy weights the ten benchmarks equally.

Figure 4: Per-GPU model-load memory for Qwen3 (235B). MXCode denotes the shared Pre/PostScale FP4 projection backend; the plotted footprint uses PostScale-style codebook. Percentages in black indicate free HBM that can be used by KV-cache. MXCode (E8M0-B32-FP4) & Vanilla VQ (BF16) has similiar footprint to MXCode.

Figure 5: Serving throughput vs. P90 interactivity for 32K-input/1K-output requests on Qwen3 (235B) using single GB300 GPU. Point labels indicate client concurrency, including queued requests. MXCode denotes the shared Pre/PostScale FP4 projection backend.

Method	P90 TTFT (s)	P90 TPOT (ms)	P90 E2E (s)
Native FP4 E4M3-B16-FP4	8.688	27.24	32.52
MXCode E4M3-B16-FP4	0.486 (-94.4%)	20.99 (-23.0%)	21.87 (-32.8%)
MXCode E8M0-B32-FP4	0.477 (-94.5%)	20.56 (-24.5%)	21.43 (-34.1%)

Table 3: 32K/1K prefix-reuse latency using one GB300 per method on Qwen3 (235B) model. Values are means of per-run 90th percentiles after post-warmup repetitions; parentheses show latency reductions relative to Native FP4. MXCode denotes the shared PreScale/PostScale FP4 projection backend.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Scale compression versus coarser scales for E4M3-B16 (top) and E8M0-B32 (bottom). Diamonds use coarser shared scales; triangles use scale VQ at matched compression ratios. Points vary QBlock size, with all representation metadata included in BPE.

	Method	MXCode PostScale			MXCode PreScale			MXCode PreScale+
	Qblock size	E5M3	E4M3	$\Delta$	E5M3	E4M3	$\Delta$	E5M3	E4M3	$\Delta$
Qwen3 (235B)	Full tensor	7.035	7.046	$-0.011$	10.349	10.359	$-0.010$	10.049	10.047	$+0.002$
	$2048\times 2048$	7.795	7.798	$-0.003$	10.363	10.361	$+0.002$	10.051	10.052	$-0.001$
	$1024\times 1024$	8.795	8.784	$+0.011$	10.384	10.380	$+0.004$	10.079	10.078	$+0.001$
	$512\times 512$	9.342	9.317	$+0.025$	10.429	10.424	$+0.005$	10.145	10.142	$+0.003$
	$256\times 256$	9.802	9.796	$+0.006$	10.572	10.571	$+0.001$	10.327	10.326	$+0.001$

Appendix

Table 4: Weight SQNR (dB; higher is better) across Qblock sizes for 300 sampled Qwen3-235B expert tensors. VQ uses the input-channel axis with

K=256

d=4

. PreScale+ compresses scales using

K_{s}=256

d_{s}=8

. E5M3 values are estimates and E4M3 values are proxies.

\Delta=\mathrm{SQNR}_{\mathrm{E5M3}}-\mathrm{SQNR}_{\mathrm{E4M3}}

, computed from displayed values; positive values favor E5M3.

Method

Native FP4

MXCode PostScale

MXCode PreScale

MXCode PreScale+

Benchmark

E5M3

E4M3

\Delta

E5M3

E4M3

\Delta

E5M3

E4M3

\Delta

E5M3

E4M3

\Delta

Weight (BPE)

4.50

–

2.06

–

2.56

–

2.13

–

MMLU-Pro

82.45

\pm 0.34

82.41

\pm 0.34

+0.04

79.04

\pm 0.36

79.18

\pm 0.36

-0.14

79.75

\pm 0.35

80.34

\pm 0.35

-0.59

80.54

\pm 0.35

80.54

\pm 0.35

0.00

RULER

95.11

\pm 0.13

95.05

\pm 0.14

+0.06

94.31

\pm 0.14

94.16

\pm 0.15

+0.15

94.89

\pm 0.14

95.01

\pm 0.14

-0.12

94.97

\pm 0.14

95.00

\pm 0.14

-0.03

DROP

88.49

\pm 0.33

88.56

\pm 0.33

-0.07

86.69

\pm 0.35

86.98

\pm 0.34

-0.29

87.71

\pm 0.34

86.54

\pm 0.35

+1.17

87.56

\pm 0.34

87.69

\pm 0.34

-0.13

GSM8K

97.35

\pm 0.44

96.89

\pm 0.48

+0.46

96.59

\pm 0.50

96.66

\pm 0.49

-0.07

97.12

\pm 0.46

97.12

\pm 0.46

0.00

96.97

\pm 0.47

96.36

\pm 0.52

+0.61

Appendix

Table 5: W4A4 evaluation on Qwen3 (235B), comparing E5M3-B16-FP4 and E4M3-B16-FP4. Accuracy and standard errors are percentages.

\Delta

denotes E5M3 minus E4M3 accuracy in percentage points; positive values favor E5M3. Weight storage includes indices, codebooks, and scales, as applicable. Average accuracy weights the ten benchmarks equally.

Quantization

See all 59 papers

XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference
May 14, 2026Thomas WittLLM Quantization Vector Quantization
Gemini Stiftung Leipzig
MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
Jul 20, 2026Simla Burcu Harma, Danila Mishin, Zhengyuan Su +7Mixed-Precision Quantization LLM Quantization
EcoCloud, EPFL · Tsinghua University · MangoBoost Inc. +2
dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats
Jun 2, 2026Giuseppe Franco, Ian Colbert, Pablo Monteagudo-Lago +2Mixed-Precision Quantization LLM Quantization
AMD

May 14, 2026J/K move · Enter open · S save

XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference

Thomas Witt

Gemini Stiftung Leipzig

LLM Quantization Vector Quantization LLM Inference Acceleration Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

We introduce XFP, a dynamic weight quantizer for LLM inference that inverts the conventional workflow: the operator specifies reconstruction quality floors on per-channel cosine similarity (one strict floor for attention and shared experts, one lazy floor for routed-expert MoE); XFP determines codebook size, outlier budget, and packing per layer automatically -- no Hessian, no calibration data, no manual bit-width selection. Each weight matrix is decomposed into a sparse fp16 outlier residual and a dense sub-byte index tensor into a per-group learned codebook. Two storage modes share one auto-select frontend and one fused decode kernel: V2 (per-channel Lloyd) and V2a (shared library of L=32 codebooks per layer). On Qwen3.5-122B-A10B under V2, XFP reaches 138 tok/s single-stream decode on workstation hardware (RTX PRO 6000 Blackwell, TP=2) at 94.49% GSM8K strict-match (3 seeds, n=3957), and is 49% faster than Marlin INT4 at TP=1. For models that do not fit in the target memory envelope, we present the H-Process: a quality-driven iteration over the two cosine thresholds that finds the operating point at which the model just fits while still producing sensible output. Three constraints define its search space: the operator-set thresholds, an OOM boundary at quantize-on-load, and a garbage boundary in generation (cosine similarity steers; benches verify). On Qwen3.5-397B-A17B (512 routed experts/layer), the H-Process fits the full expert population into 2x96 GB at ~3.4 effective bits and delivers 100.9 tok/s long-output decode at 66.72% GSM8K strict-match on the full 1319-problem set (single seed at submission; multi-seed evaluation in progress), exceeding INT4 with routed-expert pruning on memory, throughput, and accuracy simultaneously.

Abstract

Quantization

Explore similar work

XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference

MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats