Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inference

Ruiyi Ding; Jie Li; Kang He; Ziyan Liu; Chengru Song; Yuedong Xu; Yuan Cheng

cs.LGSep 29, 2026

Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inference

Authors: Ruiyi Ding, Jie Li, Kang He, Ziyan Liu, Chengru Song, Yuedong Xu, Yuan Cheng

Organizations: KlingAI Research · University of Science and Technology of China · Fudan University · Shanghai AI Incubation and Innovation Center ·

Abstract

Large language models make weight storage and memory traffic major inference costs, motivating low-precision formats that represent each weight with only a few bits. Such formats use a scale to map floating-point values into a small codebook; NVFP4 improves local range utilization by letting every 16 E2M1 weights share an E4M3 block scale. Choosing that scale is difficult in GPTQ because quantizing one column updates those that follow, so evaluating a block independently can misestimate its final reconstruction error. Large models pose a second challenge: full-precision weights, calibration activations, and second-order state cannot all remain on one accelerator, while assigning complete layers to devices leaves each time-consuming layer solve serial. We introduce Schur Replay, a scale-selection algorithm that reproduces the GPTQ updates caused by each block scale and scores the resulting block error after accounting for compensation from unquantized columns. Separately, our execution infrastructure keeps only the active layer resident, tiers activations across device, host, and disk, retires full-precision layers after export, and distributes independent output rows across tensor-parallel ranks. Together, the algorithm and infrastructure attain $99.35\%$ and $100.84\%$ question-weighted recovery from BF16 across seven benchmarks on Qwen3.5-397B-A17B and Llama-3.3-70B-Instruct. On the 397B model, the infrastructure reduces measured per-layer time by $15.17\times$ over ModelOpt and $23.14\times$ over LLM Compressor, with lower memory used per GPU.

Figures & tables

Figure 1: Schur Replay proposes a local window of exportable scales, replays each candidate from the same block-entry state, and commits the candidate minimizing

L(s)=\delta_{G}(s)^{\top}S_{G\mid P}\delta_{G}(s)

Figure 2: Scalable NVFP4 PTQ infrastructure: shared-Hessian row-parallel reconstruction (left), layer-wise activation collection and weight prefetch/retirement (center), and GPU–CPU–SSD–HDFS overflow (right).

Method	MMLU-P	GSM8K	CMath	LiveCode	MGSM	HumEval	GPQA	W. Rec. (%)
BF16	76.09	92.19	91.80	44.75	92.00	91.46	55.05	100.00
MR-GPTQ	70.99	89.54	87.89	40.75	88.40	91.46	43.43	93.86
GPTQ	73.12	91.36	92.90	41.75	91.20	85.37	45.96	96.69
ScaleSweep	73.60	92.12	91.89	41.75	90.00	93.29	47.98	97.28
Four Over Six+GPTQ	73.91	90.67	91.53	40.25	89.20	87.80	52.02	97.32
SmoothQuant+GPTQ	74.07	90.22	91.53	43.00	89.20	92.68	52.53	97.60

Table 1: W4A4 results on Qwen3.5-397B-A17B. W. Rec. is the question-weighted recovery in Equation 16 . Bold marks the best quantized method; BF16 provides the full-precision reference.

Method	MMLU-P	GSM8K	CMath	LiveCode	MGSM	HumEval	GPQA	W. Rec. (%)
BF16	55.19	93.25	86.61	31.00	92.00	84.15	37.88	100.00
MR-GPTQ	49.33	89.76	85.70	29.75	91.20	84.76	36.87	91.81
GPTQ	54.37	91.81	85.97	29.25	91.20	84.15	42.42	98.67
AWQ+GPTQ	54.24	93.18	86.70	32.00	92.80	81.71	39.90	98.85
ScaleSweep	54.53	92.72	86.34	31.50	90.00	84.15	38.38	99.03
Four Over Six+GPTQ	54.90	91.51	84.70	29.25	90.00	82.32	44.44	99.14

Table 2: W4A4 results on Llama-3.3-70B-Instruct under the same seven-dataset protocol as Table 1 .

Figure 3: NVFP4 quantization efficiency. Panels (a–b) report steady-layer compute, (c–d) full-model compute, and (e) 397B peak memory. Warm bars compare infrastructure under aligned GPTQ; diagonal hatching marks projected 397B GPTQ totals. Purple cross-hatched bars are separate measured Schur Replay runs: W4A16 on one GPU for 70B and on eight GPUs for 397B. The 397B GPTQ bars use W4A4, so the purple bars are contextual measurements rather than inputs to the infrastructure speedup ratios.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Half-width $h$	Default	Macro recovery (%)
1	no	97.70
4	yes	98.92
8	no	98.62
16	no	97.21

Appendix

Table 3: Replay-window ablation on Qwen3.5-397B-A17B W4A4.

Dataset	$n_{q}$	Weight	1q (pp)	BF16	Ours
MMLU-Pro	12,032	77.8%	0.008	9,155	9,072
GSM8K	1,319	8.5%	0.076	1,216	1,227
CMath	1,098	7.1%	0.091	1,008	1,012
LiveCodeBench	400	2.6%	0.250	179	169
MGSM-English	250	1.6%	0.400	230	232
HumanEval	164	1.1%	0.610	150	152

Appendix

Table 4: Per-dataset absolute correct counts for the controlled metric

\times

residual-construction ablation. The two panels contain the same rows and together specify every cell. Bold marks the best quantized count per dataset.

Cell	Correct	Recovery (%)	$\Delta$ questions vs. BF16
BF16	12,047	–	–
Ours (replay $\times$ Schur)	11,968	99.344	$-79$
score_rtn (static $\times$ Schur)	11,953	99.220	$-94$
win16 (replay $\times$ Schur, $h=16$ )	11,960	99.278	$-87$
metric_id (replay $\times$ Euclidean)	11,878	98.597	$-169$
pure_rtn (static $\times$ Euclidean)	11,853	98.390	$-194$

Appendix

Table 5: Question-weighted recovery and deficit relative to the 12,047-correct BF16 reference.

Method	Act-order	Rel. MSE	Correct	Macro (%)	Weighted (%)
BF16 reference	–	–	12,047	100.00	100.00
RTN + shrink search	–	0.0832	11,656	94.65	96.75
GPTQ	–	0.0480	11,908	95.48	98.84
GPTQ	✓	0.0440	11,599	94.99	96.28
Schur Replay	–	0.0379	11,997	98.49	99.59
Schur Replay	✓	0.0365	12,055	99.18	100.07

Appendix

Table 6: Weight-only NVFP4 (W4A16) on Qwen3.5-397B-A17B over the same seven datasets and 15,461 questions as the W4A4 tables. “Rel. MSE” is the relative weight-reconstruction error; “Macro” and “Weighted” are unweighted and question-weighted recovery against the 12,047-correct BF16 reference.

Schur Replay group-scale selection
$\mathbf{Input:}\;w_{0}\!\in\!\mathbb{R}^{M\times B},\ U_{GG}\!\in\!\mathbb{R}^{B\times B},\ s_{0}\!\in\!\mathcal{S}^{M},\ h,\ J=3$
$S_{G}\leftarrow(U_{GG}^{\top}U_{GG})^{-1};\hskip 9.24994pts\leftarrow s_{0}$
for $t=1,\ldots,J$ do
$Q\leftarrow\rho(w_{0}/s);\hskip 9.24994ptd\leftarrow\operatorname{diag}(QS_{G}Q^{\top})$
$s\leftarrow(QS_{G}w_{0}^{\top})_{\mathrm{diag}}/\max(d,\epsilon)$ if $d>\epsilon$
end for

Appendix

Table 10

Batched replay and group-local GPTQ commit
$\mathbf{Input:}\;W\!\in\!\mathbb{R}^{M\times N},\ U\!\in\!\mathbb{R}^{N\times N},\ \mathcal{G}=\{g,\ldots,g+B-1\},\ S_{G}\!\in\!\mathbb{R}^{B\times B},\ S\!\in\!\mathbb{R}^{M\times K}$
$W^{\mathrm{in}}\leftarrow W;\hskip 9.24994ptW_{0}\leftarrow W_{:,\mathcal{G}};\hskip 9.24994ptU_{G}\leftarrow U_{\mathcal{G},\mathcal{G}};\hskip 9.24994ptR\leftarrow W_{0}[:,\emptyset,:]\otimes\mathbf{1}_{K}$
for $j=1,\ldots,B$ do
$D_{j}\leftarrow\max((U_{G})_{j,j},\epsilon);\hskip 9.24994pt\widehat{W}_{:,:,j}\leftarrow Q_{\mathrm{NVFP4}}(R_{:,:,j};S)$
$E_{j}\leftarrow(R_{:,:,j}-\widehat{W}_{:,:,j})/D_{j}$
$R_{:,:,j+1:B}\leftarrow R_{:,:,j+1:B}-E_{j}\otimes(U_{G})_{j,j+1:B}$

Appendix

Table 11

Tiered capture and asynchronous layer residency
$\mathbf{Input:}\;\{x_{b}\}_{b=0}^{N_{b}-1}\ \text{(activation batches)},\ K_{\mathrm{gpu}},\ K_{\mathrm{host}}\ \text{(tier capacities)},\ \{\theta_{i}\}_{i=0}^{L-1}$
$\tau(b)\leftarrow\begin{cases}\mathrm{GPU},&b<K_{\mathrm{gpu}},\\ \mathrm{CPU},&K_{\mathrm{gpu}}\leq b<K_{\mathrm{gpu}}+K_{\mathrm{host}},\\ \mathrm{DISK},&\text{otherwise};\end{cases}\hskip 9.24994ptC_{b}\leftarrow\operatorname{put}_{\tau(b)}(x_{b})$
$\mathcal{H}\leftarrow\operatorname{park}_{\mathrm{CPU}}(\{\theta_{i}\});\hskip 9.24994pt\mathcal{A}\leftarrow\emptyset$
Prefetch $(i)$ : $u\leftarrow i\bmod 2$ ; $P_{u}\leftarrow\operatorname{pin}(\mathcal{H}_{i})$ ; $\widetilde{\theta}_{i}\leftarrow\operatorname{H2D}_{\mathrm{async}}(P_{u})$ ; record event $\eta_{i}$
Onload $(i)$ : $\operatorname{wait}(\eta_{i})$ ; $\theta_{i}\leftarrow\widetilde{\theta}_{i}$ ; $\mathcal{A}\leftarrow\mathcal{A}\cup\{i\}$
Release $(i)$ : $\theta_{i}\leftarrow\operatorname{offload}(\theta_{i})$ or $\emptyset$ after $\operatorname{pack}(i)$ ; $\mathcal{A}\leftarrow\mathcal{A}\setminus\{i\}$

Appendix

Table 12

Recoverable layer-wise PTQ pipeline
$\mathbf{Input:}\;\{\ell_{i}=(\theta_{i},\cdot)\}_{i=0}^{L-1},\ C_{0}\ \text{(input cache)},\ w\ \text{(commit window)}$
$\mathcal{M}\ \text{(manifest)};\hskip 9.24994pti_{0}\leftarrow\operatorname{resume}(\mathcal{M});\hskip 9.24994pt\mathcal{F}_{r}\leftarrow\emptyset$ on every rank $r$
$\operatorname{park}(\{\ell_{i}\});\hskip 9.24994pt\operatorname{prefetch}(i_{0})$
for $i=i_{0},\ldots,L-1$ do
$(z_{<i-1},z_{i-1},z_{i})\leftarrow(\textsc{Skip},\textsc{Run},\textsc{Capture})$
$X_{i}\leftarrow\operatorname{materialize}(C_{i})$ ; $\operatorname{onload}(i)$ ; $\operatorname{prefetch}(i+1)$

Appendix

Table 13

Row-parallel GPTQ solve
$\mathbf{Input:}\;N_{r}=N/P_{\mathrm{tp}},\ B\mid N_{r},\ \mathcal{J}_{r}=[rN_{r},(r+1)N_{r}),\ \mathcal{B}_{r}=[rN_{r}/B,(r+1)N_{r}/B),\ r\in[P_{\mathrm{tp}}]$
$\lambda\geq 0\ \text{(absolute Hessian damping)};\hskip 9.24994ptB=16$
$W_{r}\!\in\!\mathbb{R}^{M\times N_{r}},\hskip 9.24994ptA_{r}\!\in\!\mathbb{R}^{M\times N_{r}/B}\ \text{(block-scale metadata)},\hskip 9.24994ptX_{r}\!\in\!\mathbb{R}^{N_{r}\times T}$
$X\leftarrow\operatorname{cat}_{0}(\operatorname{AG}(X_{r}))\in\mathbb{R}^{N\times T}$ // gather contraction features; retain tokens
$H_{r}\leftarrow X_{r}X^{\top}\in\mathbb{R}^{N_{r}\times N}$ // local Hessian row block
$\{H_{p}\}_{p=0}^{P_{\mathrm{tp}}-1}\leftarrow\operatorname{AG}(H_{r})$ // all-gather row blocks to every rank

Appendix

Table 14

Dataset	BF16	SOAR	Schur Replay
MMLU-Pro (12,032)	9,155	8,970	9,072
GSM8K (1,319)	1,216	1,214	1,227
CMath (1,098)	1,008	1,019	1,012
LiveCodeBench (400)	179	165	169
MGSM-English (250)	230	223	232
HumanEval (164)	150	152	152

Appendix

Table 7: Exact correct-question counts for Qwen3.5-397B-A17B. SOAR is the strongest external quantized baseline by question-weighted recovery.

Dataset	BF16	SOAR	Schur Replay
MMLU-Pro (12,032)	6,641	6,699	6,730
GSM8K (1,319)	1,230	1,216	1,229
CMath (1,098)	951	942	946
LiveCodeBench (400)	124	120	117
MGSM-English (250)	230	228	230
HumanEval (164)	138	135	137

Appendix

Table 8: Exact correct-question counts for Llama-3.3-70B-Instruct under the same protocol as Table 7 .

Method	Qwen Macro	Qwen W. Rec.	Llama Macro	Llama W. Rec.
BF16	77.62	100.00	68.58	100.00
GPTQ	74.52	96.69	68.45	98.67
SmoothQuant+GPTQ	76.18	97.60	68.15	99.26
AWQ+GPTQ	76.65	97.69	68.65	98.85
Four Over Six+GPTQ	75.05	97.32	68.16	99.14
SOAR+GPTQ	76.80	98.38	67.27	100.20

Appendix

Table 9: Seven-dataset macro mean and question-weighted recovery (%). Macro averages datasets uniformly; W. Rec. weights their question counts and normalizes by the corresponding BF16 total. Bold marks the best quantized method in each column.

Figure 4: Exhaustive audit of the Schur Replay window over all 126 exportable scales. Percentages exclude flat rows for which every candidate ties. (a) Signed distance from the proposal center to the nearest optimum of the conditional replay surrogate. (b) Fraction of rows whose local window contains such an optimum. (c) Layer-wise coverage at

h=4

. The audit contains 143.5M live rows from Qwen3.5-397B-A17B and 85.5M from Llama-3.3-70B-Instruct.

Figure 5: Scaling of a mock Qwen3.5 transformer layer with Qwen3.5-397B-A17B layer width under TP8 as calibration blocks increase at sequence length 16,384. The node has eight 72 GB GPUs, 3.5 TB of local SSD capacity, and mounted HDFS storage. (a) Scaling curves show completed elapsed-time measurements. (b) Grouped bars show the same measurements for direct comparison at each workload size. Both panels use logarithmic time axes; crosses mark runs that did not complete under host-memory pressure and are not assigned synthetic times or bar heights. Ours is faster for every jointly completed setting and is the only evaluated path that completes the 8,192-block stress test.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

Aug 28, 2026

H-Scale: Hessian-Guided Scale Refinement for NVFP4 Sub-Byte LLM Inference

The NVIDIA Blackwell architecture, with native support for the ultra-fine-grained NVFP4 format, opens new opportunities for accelerating large language model (LLM) inference. NVFP4's micro-block design, such as a group size of 16, offers strong representational flexibility for capturing local weight distributions and isolating outliers, but it also introduces a large and highly sensitive space of per-group scaling factors. Existing post-training quantization (PTQ) methods primarily focus on refining quantized weight values, leaving this scale-selection step underexplored. To address this gap, we propose H-Scale, a lightweight post-processing method for NVFP4 per-group scale refinement. Instead of minimizing plain weight reconstruction error, H-Scale selects hardware-valid group scales using a diagonal second-order proxy derived from calibration activations, thereby targeting layer output perturbation more directly. It is designed as a drop-in replacement for RTN-style scale selection in diverse NVFP4 pipelines, requires only modest offline calibration, and introduces strictly zero overhead at inference time. Under a fixed evaluation protocol, experiments on mainstream LLMs show that H-Scale generally improves a broad range of NVFP4 baselines and brings several variants closer to the BF16 reference.

Hao Yu, Zheng Li, Dayiheng Liu +1

Qwen Team, Alibaba Inc.

LLM Quantization Neural Network Quantization Post-Training Quantization Low-Bit Quantization

Ask PDF

May 12, 2026

Search Your Block Floating Point Scales!

Quantization has emerged as a standard technique for accelerating inference for generative models by enabling faster low-precision computations and reduced memory transfers. Recently, GPU accelerators have added first-class support for microscaling Block Floating Point (BFP) formats. Standard BFP algorithms use a fixed scale based on the maximum magnitude of the block. We observe that this scale choice can be suboptimal with respect to quantization errors. In this work, we propose ScaleSearch, an alternative strategy for selecting these scale factors: using a fine-grained search leveraging the mantissa bits in microscaling formats to minimize the quantization error for the given distribution. ScaleSearch can be integrated with existing quantization methods such as Post Training Quantization and low-precision attention, and is shown to improve their performance. Additionally, we introduce ScaleSearchAttention, an accelerated NVFP4-based attention algorithm, which uses ScaleSearch and adapted prior techniques to ensure near-0 performance loss for causal language modeling. Experiments show that ScaleSearch reduces quantization error by 27% for NVFP4 and improves language model PTQ by up to 15 points for MATH500 (Qwen3-8B), while ScaleSearchAttention improves Wikitext-2 PPL by upto 0.77 points for Llama 3.1 70B. The proposed methods closely match baseline performance while providing quantization accuracy improvements.

Tanmaey Gupta, Hayden Prairie, Xiaoxia Wu +10

Cornell University, Ithaca, NY, USA · Together AI, USA · University of California San Diego, La Jolla, CA, USA +1

4-Bit Quantization LLM Inference Acceleration Post-Training Quantization Neural Network Quantization

Ask PDF

May 30, 2026

ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization

NVFP4 is a recently introduced hardware-supported FP4 format that improves the fidelity of 4-bit quantization through fine-grained block scales. However, existing NVFP4 scale initialization methods still primarily rely on AbsMax initialization, which leaves a noticeable gap to the optimal solution. To address this, we propose ScaleSweep, a simple and efficient scale optimization method that sweeps over feasible block scale candidates and selects the candidate that minimizes a target objective. We further provide a theoretical analysis of NVFP4 quantization and derive both lower and upper bounds for the required sweep range under mean square error (MSE) and weighted mean square error (WMSE) between the original tensor and the quantized reconstructed tensor. The proposed bounds substantially reduce the sweep space while preserving the optimal candidate, enabling negligible overhead compared with the baseline quantization operators. Experiments on Llama and Qwen models demonstrate that ScaleSweep consistently improves quantization performance over existing initialization methods and further narrows the gap to full precision. In particular, under aggressive end-to-end quantization of weights, activations, KV cache, and query states, ScaleSweep preserves more than 93% of the full-precision performance.

Li Lin, Xiaojun Wan

Wangxuan Institute of Computer Technology, Peking University

LLM Quantization 4-Bit Quantization Activation Quantization Post-Training Quantization Neural Network Quantization

Ask PDF

Abstract

LLM Quantization

Explore similar work

H-Scale: Hessian-Guided Scale Refinement for NVFP4 Sub-Byte LLM Inference

Search Your Block Floating Point Scales!

ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization