Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity

Hengyuan Zhang; Xinrong Chen; Zunhai Su; Xiao Liang; Jing Xiong; Wendong Xu; He Xiao; Chaofan Tao

cs.LGMar 18, 2026

Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity

Authors: Hengyuan Zhang, Xinrong Chen, Zunhai Su, Xiao Liang, Jing Xiong, Wendong Xu, He Xiao, Chaofan Tao,

Organizations: The University of Hong Kong · Peking University · Tsinghua University · University of California, Los Angeles ·

Abstract

Layer-wise mixed-precision quantization (LMPQ) enables effective compression under extreme low-bit settings by allocating higher precision to sensitive layers. However, existing methods typically treat all intra-layer weight modules uniformly and rely on a single numerical property when estimating sensitivity, overlooking their distinct operational roles and structural characteristics. To address this, we propose NSDS, a novel calibration-free LMPQ framework driven by Numerical and Structural Dual-Sensitivity. Specifically, it first mechanistically decomposes each layer into distinct operational roles and quantifies their sensitivity from both numerical and structural perspectives. These dual-aspect scores are then aggregated into a unified layer-wise metric through a robust aggregation scheme based on MAD-Sigmoid and Soft-OR to guide bit allocation. Extensive experiments demonstrate that NSDS consistently achieves superior performance compared to various baselines across diverse models and downstream tasks, without relying on any calibration data.

Figures & tables

Figure 1: Layer-wise sensitivity across two LLMs. Each dot represents a layer. Darker dots indicate more severe perplexity degradation (

\Delta

PPL) when quantizing that layer. See § 2.2 for the calculation of Numerical Vulnerability and Structural Expressiveness. Red dashed boxes highlight that layers with low numerical but high structural sensitivity still suffer significant degradation. See Appendix A for more results.

Figure 2: Overview of our NSDS framework for data-free layer sensitivity estimation. The left panel illustrates the mechanistic decomposition of a layer into distinct operational components, categorized as Detectors and Writers (§ 2.1 ). The middle panel presents the dual-view sensitivity estimation: Numerical Vulnerability (NV) and Structural Expressiveness (SE) (§ 2.2 ). The right panel shows the robust aggregation process, where MAD-Sigmoid normalizes heterogeneous scores and Soft-OR integrates them into a unified layer-wise sensitivity metric, which is then used for bit allocation under a target budget (§ 2.3 ). See Appendix B for the algorithmic description of the NSDS framework.

	Language Reasoning ( $\uparrow$ )						Language Modeling ( $\downarrow$ )
	ARC-C	Hellaswag	PIQA	BoolQ	Winogrande	TruthfulQA	Wikitext-2	C4
	Llama-3.1-8B
FP16	57.76	81.97	80.09	82.11	77.35	28.40	6.24	8.95
[1pt/1pt] MSE	38.73	64.21	68.58	60.73	65.82	23.95	9.22	12.35
EWQ	40.17	67.16	71.49	64.52	67.43	23.34	8.38	11.27
ZD	39.45	66.94	69.63	62.88	67.51	24.49	8.42	11.51

Table 1: Results of baseline methods and our NSDS on Llama-3.1-8B and Qwen2.5-7B, evaluated on six language reasoning and two language modeling benchmarks. See Appendix E for the details of baseline methods.

Figure 3: Average accuracy of NSDS and baselines on language reasoning benchmarks across different bit budgets for Llama-3.1-8B and Qwen2.5-7B.

	Llama-2-13B		Qwen2.5-14B
	Acc. ( $\uparrow$ )	PPL ( $\downarrow$ )	Acc. ( $\uparrow$ )	PPL ( $\downarrow$ )
FP16	67.23	5.67	73.25	7.22
[1pt/1pt] MSE	59.91	9.38	64.29	10.62
EWQ	61.68	7.61	63.36	11.13
ZD	60.99	7.98	65.42	10.25
KurtBoost	62.08	7.51	65.13	10.41

Table 2: Average PPL and accuracy of NSDS and baselines on language modeling and reasoning benchmarks for larger-scale LLMs. See Table 3 for detailed results.

Figure 4: Average accuracy of the ablation analysis on NSDS. “w/o” denotes the exclusion of a specific component from NSDS during layer sensitivity estimation.

Figure 5: Average accuracy comparison between NSDS framework and calibration-based baselines across generic reasoning benchmarks.

Figure 6: Average accuracy on language reasoning benchmarks when integrating NSDS with different PTQ backends, compared against the SliM-LLM.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visualization of NV, SE, and NSDS scores across layers on Llama-3.1-8B and Qwen2.5-7B. Darker colors indicate higher sensitivity score.

	Language Reasoning ( $\uparrow$ )						Language Modeling ( $\downarrow$ )
	ARC-C	Hellaswag	PIQA	BoolQ	Winogrande	TruthfulQA	Wikitext-2	C4
	Llama-2-13B
FP16	59.47	82.22	79.05	80.55	76.16	25.95	4.88	6.47
[1pt/1pt] MSE	46.71	73.27	73.35	72.58	71.63	21.96	8.55	10.21
EWQ	48.14	75.11	75.38	75.22	73.43	22.85	6.77	8.43
ZD	47.89	74.34	74.83	74.01	72.69	22.23	7.08	8.89

Appendix

Table 3: Results of baseline methods and our NSDS on Llama-2-13B and Qwen2.5-14B, evaluated on six language reasoning and two language modeling benchmarks.

Figure 8: Average perplexity results of the ablation analysis on NSDS across all language modeling benchmarks. “w/o” denotes the exclusion of a specific component from NSDS during layer sensitivity estimation.

Figure 9: Average perplexity comparison between NSDS framework and calibration-based baselines across generic reasoning benchmarks.

Figure 10: Average perplexity on language modeling benchmarks when integrating NSDS with different PTQ backends, compared against the SliM-LLM.

Mixed-Precision Quantization

Mixed-precision quantization assigns different numeric precisions across a model to balance accuracy with memory, speed, energy, and hardware constraints.

See all 168 papers

Beyond Scalar Sensitivity: Activation-Aware Mixed-Precision LLM Quantization with Cross-Layer Refinement
Sep 22, 2026Akihiro Yoshida, Yuma IchikawaMixed-Precision Quantization LLM Quantization
Fujitsu Limited, Institute of Science Tokyo · Fujitsu Limited, RIKEN Center for AIP
MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
Jul 25, 2026Ashitabh Misra, Madhav Agrawal, Arham Jain +1Mixed-Precision Quantization LLM Quantization
University of Illinois at Urbana-Champaign
Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantization
Jul 1, 2026Fei Wang, Chao Xue, Taoran Liu +3Mixed-Precision Quantization LLM Quantization
South China University of Technology · Ocean University of China · Shenzhen Campus of Sun Yat-sen University +1

Sep 22, 2026J/K move · Enter open · S save

Beyond Scalar Sensitivity: Activation-Aware Mixed-Precision LLM Quantization with Cross-Layer Refinement

Akihiro Yoshida, Yuma Ichikawa

Fujitsu Limited, Institute of Science Tokyo · Fujitsu Limited, RIKEN Center for AIP

Mixed-Precision Quantization LLM Quantization Weight-Only Quantization Neural Network Quantization Quantization Sensitivity Analysis

Open paper AskPDF arXiv

Mixed-precision weight quantization is commonly formulated as a Multiple-Choice Knapsack Problem (MCKP), yet existing solvers rely on scalar sensitivity proxies that collapse each weight matrix's Hessian into a single number and treat every module independently. We prove that even the optimal scalar proxy incurs multiplicative distortion up to

\sqrt{κ(\mathbf{A})κ(\mathbf{B})}

relative to the full activation-aware quadratic, where

κ(\mathbf{A})

and

κ(\mathbf{B})

denote the condition numbers of the input- and output-side Hessian factors. This bound varies from

10^1

10^{13}

for typical LLM modules, making inter-module sensitivity ranking unreliable. To address these limitations, we propose Cross-layer Activation-aware Sensitivity Allocation (CASA), a two-phase method. In Stage 1, the scalar proxy is replaced by an activation-aware metric derived from the Kronecker-factored Hessian, reducing the MCKP to a form whose continuous relaxation admits a closed-form solution. In Stage 2, a cross-layer-aware local search evaluates bit-width updates using the end-to-end model loss. Experiments on multiple LLMs across different bit budgets show that CASA achieves lower perplexity than the latest scalar-proxy baselines, especially at ultra-low bit-widths (

<3

bits per weight). Moreover, the performance gain in zero-shot accuracy tracks the per-model average condition-number over modules, confirming the distortion bound as a practical indicator of scalar-proxy failure.

Abstract

Mixed-Precision Quantization

Explore similar work

Beyond Scalar Sensitivity: Activation-Aware Mixed-Precision LLM Quantization with Cross-Layer Refinement

MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantization