Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4

Musa Cim; Burak Topcu; Mahmut Taylan Kandemir

cs.ARMar 5, 2026

Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4

Authors: Musa Cim, Burak Topcu, Mahmut Taylan Kandemir

Organizations: Department of Computer Science and Engineering The Pennsylvania State University University Park, PA, USA

Abstract

Quantization addresses the high resource demand for large language models (LLMs) by alleviating memory pressure and bandwidth congestion and providing significantly scaled compute power with a tolerable impact on accuracy. Four-bit floating point (FP4), the lowest-precision format that preserves essential numerical properties such as exponent and sign, has begun to be adopted in cutting-edge architectures, including Blackwell and AMD CDNA, to support LLM quantization and reduce deployment costs. Although aggressive quantization can yield efficiency gains, the quantization sensitivity of within-transformer layers and whether these sensitivities generalize across existing FP4 formats and model scales remain underexplored. To elucidate quantization sensitivity, this study conducts a systematic analysis of two FP4 formats, MXFP4 and NVFP4, across three Qwen2.5 model scales (0.5B, 7B, and 14B), using controlled component-wise and block-wise isolation methodologies. We observe that MLP up- and down-projection layers consistently dominate in terms of sensitivity, while gate and attention projections are moderately and substantially less sensitive to FP4 quantization, respectively. We further find that sensitivity does not universally localize to the final blocks, but early blocks can be highly sensitive, particularly under MXFP4. Our results provide a diagnostic characterization of the inference behavior of FP4 across components, depths, and FP4 formats.

Figures & tables

Panel of Figure 1 — Figure 1: Component sensitivity comparison across three model scales. Rows: 0.5B, 7B, 14B. Blue = MLP, Red = Attention. MLP projections (down and up) consistently form the most sensitive tier across all scales and formats.

Figure 2: Block-wise sensitivity analysis for down projection across three model scales. Y-axis uses symlog scale. Positive values indicate PPL improvement when keeping that block in FP16.

	Qwen2.5-0.5B			Qwen2.5-7B			Qwen2.5-14B
Component	P99.9	Max	Ratio	P99.9	Max	Ratio	P99.9	Max	Ratio
down_proj	2.8	253	159	4.3	310	80	4.2	300	334
up_proj	11.2	58	5.5	9.9	49	5.5	5.1	42	10
gate_proj	11.2	58	5.5	9.9	49	5.5	5.1	46	11
v_proj	13.0	36	2.9	7.0	30	4.4	4.8	23	4.8
o_proj	2.4	6	2.6	2.4	9	3.4	1.8	6	3.6

Table 1: Activation statistics by component across all model scales. All values are averaged across layers. The Max/P99.9 ratio is computed as the mean of per-layer ratios (not the ratio of averaged Max to averaged P99.9), which better captures outlier severity across depth. Down projection, as highlighted, consistently shows substantially worse ratios than other components.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

	NVFP4		MXFP4
Component	PPL	Improvement	PPL	Improvement
Baseline	21.63	—	36.71	—
up_proj	20.40	+1.23	28.46	+8.25
down_proj	20.47	+1.16	28.79	+7.92
gate_proj	20.73	+0.90	31.24	+5.47
v_proj	20.99	+0.64	31.45	+5.26

Appendix

Table 2: Component sensitivity under NVFP4 and MXFP4 for Qwen2.5-0.5B. PPL improvement measures sensitivity (higher = keeping that component in FP16 helps more).

	NVFP4		MXFP4
Component	PPL	Improvement	PPL	Improvement
Baseline	6.47	—	7.31	—
down_proj	6.36	+0.11	6.81	+0.50
up_proj	6.37	+0.10	6.99	+0.31
gate_proj	6.39	+0.08	7.08	+0.23
o_proj	6.43	+0.04	7.18	+0.13

Appendix

Table 3: Component sensitivity under NVFP4 and MXFP4 for Qwen2.5-7B.

	NVFP4		MXFP4
Component	PPL	Improvement	PPL	Improvement
Baseline	5.26	—	6.02	—
down_proj	5.10	+0.16	5.63	+0.39
up_proj	5.12	+0.14	5.77	+0.24
gate_proj	5.14	+0.12	5.82	+0.20
o_proj	5.21	+0.05	5.89	+0.13

Appendix

Table 4: Component sensitivity under NVFP4 and MXFP4 for Qwen2.5-14B.

Figure 3: Block sensitivity heatmaps showing PPL improvement when each block’s component is kept in FP16. Left: MXFP4 (scale 0–2.0). Right: NVFP4 (scale 0–0.28). Note the 7 $\times$ scale difference and different spatial patterns.

Figure 4: Block sensitivity for up_proj . MXFP4 shows strong early-block sensitivity (blocks 0, 8, 23), while NVFP4 peaks at block 23.

Figure 5: Percentage change from baseline for up_proj . Block 23 shows $-3.3\%$ (MXFP4) and $-1.3\%$ (NVFP4) improvement.

Figure 6: Block sensitivity for down_proj (0.5B only). Both formats show block 23 as highly sensitive, but MXFP4 also shows strong sensitivity in blocks 0–3.

Figure 7: Percentage change from baseline for down_proj . Block 23 shows $-3.3\%$ (MXFP4) and $-1.3\%$ (NVFP4) improvement.

Figure 8: Block sensitivity for gate_proj . MXFP4 exhibits pronounced early-block dominance (blocks 0–2), while NVFP4 shows late-block sensitivity peaking at block 23.

Figure 9: Percentage change from baseline for gate_proj .

Figure 10: Block sensitivity for o_proj . MXFP4 shows extreme sensitivity in block 1 (+2.14 PPL improvement), a unique pattern not seen in other components.

Figure 11: Percentage change from baseline for o_proj . Block 1 shows $-5.8\%$ improvement for MXFP4.

Figure 12: Block sensitivity for v_proj . MXFP4 shows early-block dominance, while NVFP4 exhibits relatively flat sensitivity with a slight peak at block 8.

Figure 13: Percentage change from baseline for v_proj .

Figure 14: Block sensitivity for k_proj . Both formats show relatively low and uniform sensitivity, consistent with k_proj being the second-least sensitive component.

Figure 15: Percentage change from baseline for k_proj .

Figure 16: Block sensitivity for q_proj . The least sensitive component overall, showing minimal variation across blocks for both formats.

Figure 17: Percentage change from baseline for q_proj .

Figure 18: Block sensitivity heatmaps for Qwen2.5-7B showing PPL improvement when each block’s component is kept in FP16. Left: MXFP4. Right: NVFP4. Block 27 shows strong sensitivity for MLP components in both formats.

Figure 19: Block sensitivity for down_proj (7B). Both formats show block 27 as highly sensitive.

Figure 20: Percentage change from baseline for down_proj (7B). Block 27 shows $-3.0\%$ (MXFP4) and $-0.47\%$ (NVFP4) improvement.

Figure 21: Block sensitivity for up_proj (7B). Block 27 is most sensitive for both formats.

Figure 22: Percentage change from baseline for up_proj (7B). Block 27 shows $-1.2\%$ (MXFP4) and $-0.37\%$ (NVFP4) improvement.

Figure 23: Block sensitivity for gate_proj (7B). Block 27 shows highest sensitivity.

Figure 24: Percentage change from baseline for gate_proj (7B). Block 27 shows $-0.76\%$ (MXFP4) and $-0.19\%$ (NVFP4) improvement.

Figure 25: Block sensitivity for o_proj (7B). MXFP4 shows early-block sensitivity, while NVFP4 shows relatively flat sensitivity.

Figure 26: Percentage change from baseline for o_proj (7B). MXFP4 shows $\sim$ 0.3% variation, while NVFP4 shows $<$ 0.05%.

Figure 27: Block sensitivity for v_proj (7B). MXFP4 shows block 27 as most sensitive, while NVFP4 exhibits relatively flat sensitivity.

Figure 28: Percentage change from baseline for v_proj (7B).

Figure 29: Block sensitivity for k_proj (7B). Both formats show relatively low and uniform sensitivity across blocks.

Figure 30: Percentage change from baseline for k_proj (7B).

Figure 31: Block sensitivity for q_proj (7B). MXFP4 shows block 1 as most sensitive, while NVFP4 shows minimal variation across blocks.

Figure 32: Percentage change from baseline for q_proj (7B).

Quantization Sensitivity Analysis

See all 20 papers

MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
Jul 20, 2026Simla Burcu Harma, Danila Mishin, Zhengyuan Su +7Mixed-Precision Quantization LLM Quantization
EcoCloud, EPFL · Tsinghua University · MangoBoost Inc. +2
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
Jan 12, 2026Haoqian Meng, Yilun Luo, Yafei Zhao +3LLM Quantization 4-Bit Quantization
School of Computer Science and Technology, Tianjin University
FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling
Aug 3, 2026Xianglong Yan, Hong Liu, Chengzhu Bao +4LLM Quantization 4-Bit Quantization
Shanghai Jiao Tong University · Tencent

Jul 20, 2026J/K move · Enter open · S save

MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

Simla Burcu Harma, Danila Mishin, Zhengyuan Su, Ayan Chakraborty+6

EcoCloud, EPFL · Tsinghua University · MangoBoost Inc.+2

Mixed-Precision Quantization LLM Quantization LLM Inference Acceleration Post-Training Quantization Neural Network Quantization Quantization Sensitivity Analysis

Open paper AskPDF arXiv

4-bit quantization enables efficient LLM inference, but suffers from significant accuracy degradation due to outliers. Prior work addresses this problem via data rotation or mixed-precision integer quantization, but often relies on software-managed scaling and frequent dequantization, incurring substantial overhead. Microscaling formats, such as MXINT, eliminate these inefficiencies by encoding scales in hardware, yet remain incompatible with rotation-based methods. Our analysis reveals that outliers vary in severity, from rare extremes to frequent mild deviations, and that quantization sensitivity is unevenly distributed across layers and columns. These insights motivate a fine-grained, sensitivity-guided approach. We introduce MXSens, a training-free method that assigns mixed mantissa bitwidths (4/6/8) based on column- and layer-wise sensitivity, naturally leveraging the block-wise structure of MXINT. MXSens outperforms state-of-the-art quantization methods across a range of models and tasks. Under the W4A4KV4 setting, MXSens achieves perplexities of 3.77 and 7.63 on LLaMA-2-70B and LLaMA-3-8B, respectively, substantially improving over existing baselines on WikiText-2. Our work establishes a new balance between accuracy and resource efficiency for LLM quantization.

Abstract

Quantization Sensitivity Analysis

Explore similar work

MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling