OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization

Neriah Ben David; Ori Meir; Or Ordentlich

cs.AIOct 6, 2026

OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization

Authors: Neriah Ben David, Ori Meir, Or Ordentlich

Abstract

NVFP4 is an attractive datatype for large language model (LLM) inference, offering compact storage and native tensor-core acceleration. However, preserving accuracy using NVFP4 requires careful quantization. In this work we develop a novel quantization scheme called Optimized Smoothing and Scaling for NVFP4 (OSFP4). For each linear projection it uses a diagonal smoothing matrix whose entries are optimized to minimize the squared matrix-product quantization error under NVFP4, taking into account the rounding procedure that is used (either round-to-nearest, or GPTQ-style successive interference cancellation). This requires performing joint optimization on the smoothing entries as well as the block scales, which is facilitated by analyzing a multiplicative-dither FP4 quantizer instead of the fixed deterministic one. Experiments show that OSFP4 achieves the highest average accuracy among the evaluated competitors in the corresponding quantization settings, while retaining approximately 94-97% of vendor NVFP4 prefill throughput on the measured workloads. Our code is available in https://github.com/neriahbd/OSFP4

Explore similar work

SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization
May 12, 2026Chengzhu Bao, Xianglong Yan, Zhiteng Li +3LLM Quantization 4-Bit Quantization
Shanghai Jiao Tong University · Tencent Hunyuan
ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization
May 30, 2026Li Lin, Xiaojun WanLLM Quantization 4-Bit Quantization
Wangxuan Institute of Computer Technology, Peking University
FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling
Aug 3, 2026Xianglong Yan, Hong Liu, Chengzhu Bao +4LLM Quantization 4-Bit Quantization
Shanghai Jiao Tong University · Tencent

May 12, 2026, cs.LGJ/K move · Enter open · S save

SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization

Chengzhu Bao, Xianglong Yan, Zhiteng Li, Guangshuo Qin+2

Shanghai Jiao Tong University · Tencent Hunyuan

LLM Quantization 4-Bit Quantization Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

NVFP4 has recently emerged as an efficient 4-bit microscaling format for large language models (LLMs), offering superior numerical fidelity with native hardware support. However, existing methods often yield suboptimal performance due to inflexible scale selection and the coupled treatment of quantization and dequantization scales. To address these issues, we propose Scale Optimization for Accurate Reconstruction (SOAR), a novel post-training quantization framework that improves the accuracy of NVFP4 quantization. At its core, SOAR features Closed-form Joint Scale Optimization (CJSO), which jointly optimizes global and block-wise scales via analytical solutions derived from reconstruction error minimization. Furthermore, it incorporates Decoupled Scale Search (DSS). DSS decouples the high-precision quantization scale from its constrained dequantization counterpart, and performs discrete search to mitigate precision loss from scale quantization. Extensive experiments across multiple LLMs show that our method consistently outperforms existing NVFP4 quantization baselines, achieving superior accuracy under the same memory footprint with no additional hardware overhead. The code and models will be available at https://github.com/steven-bao1/SOAR.

Abstract

Explore similar work

SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization

ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization

FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling