4-Bit Quantization

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 65

All topics

APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing
Jun 7, 2026Hong Guo, Nianhui Guo, Weixing Wang +3LLM Quantization GPU Kernel Optimization
Hasso Plattner Institute · German University of Digital Science
Trainable Smooth-Rotation Transforms with Learned Channel Scales for LLM Quantization
Jun 7, 2026Patrik Czakó, Gábor Kertész, Sándor SzénásiLLM Quantization 4-Bit Quantization
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University Budapest, Hungary · John von Neumann Faculty of Informatics, Obuda University Budapest, Hungary
OffQ: Taming Structured Outliers in LLM Quantization by Offsetting
Jun 5, 2026Haoqi Wang, Lorenz K. Mueller, Jiawei Zhuang +2LLM Quantization 4-Bit Quantization
School of Computer and Communication Sciences, EPFL, Switzerland · 2Huawei, Switzerland · 3Swiss Data Science Center, ETHZ & EPFL, Switzerland
Characterizing the Impact of NVFP4 Quantization for Low-Power Edge AI Deployment
Jun 3, 2026Ovishake Sen, Venkata Nithin Kamineni, Daniel Lobo +3Efficient Neural Network Inference Edge Inference
Department of Electrical and Computer Engineering, University of Florida, Gainesville, USA
MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models
Jun 3, 2026Yue Wu, Changyuan Wang, Zixuan Wang +2LLM Quantization 4-Bit Quantization
ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization
May 30, 2026Li Lin, Xiaojun WanLLM Quantization 4-Bit Quantization
Wangxuan Institute of Computer Technology, Peking University
Not All NVFP4 QAT Recipes Are Equal: How Architecture and Scale Shape Model Quality for Anomaly Segmentation
May 26, 2026Zijian Du, Oleg RybakovEfficient Neural Network Inference Image Segmentation
NVIDIA
Timestep-Aware SVDQuant-GPTQ for W4A4 Quantization of Wan2.2-I2V
May 26, 2026Junhao Wu, Dezhong Yao, Hai JinVideo Diffusion Models Diffusion Transformer
School of Computer Science and Technology, Huazhong University of Science and Technology Wuhan, Hubei, China 430074
Tail-Aware HiFloat4: W4A4 Post-Training Quantization for Wan2.2
May 26, 2026Zhanfeng Feng, Shuai Guo, Xin Di +3Video Diffusion Models 4-Bit Quantization
University of Science and Technology of China Hefei, China
Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit
May 25, 2026Zexin Zhuang, Yanhang Li, Zhichao FanLLM Quantization Statistical Power Analysis
1Southern Methodist University · 2Northeastern University · University of Illinois Urbana-Champaign
Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floor
May 19, 2026Xiaocan Li, Shiliang Wu, Zheng ShenLLM Quantization 4-Bit Quantization
Huawei Canada
CAR-SAM: Cross-Attention Reconstruction for Post-Training Quantization of the Segment Anything Model
May 16, 2026Houji Wen, Jiangyong Yu, Jun Li +1Cross-Attention 4-Bit Quantization
1Nanjing University of Science and Technology · 2Houmo AI
E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring
May 16, 2026Wenjun Wang, Yanggan Gu, Shuo Cai +44-Bit Quantization Post-Training Quantization
The Hong Kong Polytechnic University · PolyU-Daya Bay Technology and Innovation Research Institute
DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers
May 16, 2026Sayeh Sharify, Mahsa Salmani, Hesham MostafaDiffusion Transformer Diffusion Model Quantization
d-Matrix Santa Clara, CA 95054
Search Your Block Floating Point Scales!
May 12, 2026Tanmaey Gupta, Hayden Prairie, Xiaoxia Wu +104-Bit Quantization LLM Inference Acceleration
Cornell University, Ithaca, NY, USA · Together AI, USA · University of California San Diego, La Jolla, CA, USA +1
Grid Games: The Power of Multiple Grids for Quantizing Large Language Models
May 12, 2026Vage Egiazarian, Erik Schultheis, Andrei Panferov +3LLM Quantization 4-Bit Quantization
ISTA · Aril Computer · ETH Zürich +1
SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization
May 12, 2026Chengzhu Bao, Xianglong Yan, Zhiteng Li +3LLM Quantization 4-Bit Quantization
Shanghai Jiao Tong University · Tencent Hunyuan
MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
May 12, 2026Yupeng Su, Ruijie Zhang, Ziyue Liu +24-Bit Quantization Muon Optimizer
University of California, Santa Barbara
AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
May 9, 2026Beshr IslamBouli, David JinLLM Quantization 4-Bit Quantization
Massachusetts Institute of Technology, Cambridge, MA, USA.
Finer is Better (with the Right Scaling)
May 8, 2026Clemens Schaefer, Gil TabakLLM Quantization 4-Bit Quantization
Google LLC, Mountain View, Ca
Direction-Preserving Number Representations
May 8, 2026Bardia Zadeh, George A. Constantinides4-Bit Quantization Low-Bit Quantization
Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom
Normalized Architectures are Natively 4-Bit
May 7, 2026Maxim Fishman, Brian Chmiel, Ron Banner +2LLM Quantization Neural Representation Geometry
∧NVIDIA · ◦Department of Electrical and Computer Engineering - Technion, Haifa, Israel
When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon
May 7, 2026Mohamed Amine BergachGPU Kernel Optimization KV Caching
Illumina, San Diego, CA, USA
BitCal-TTS: Bit-Calibrated Test-Time Scaling for Quantized Reasoning Models
May 7, 2026Sai Babu Patarlapalli, Surya Teja AvvaruLLM Inference 4-Bit Quantization
DurableUn: Quantization-Induced Recovery Attacks in Machine Unlearning
May 4, 2026Abdullah Ahmad Khan, Ferdous SohelQuantization-Aware Training 4-Bit Quantization
Murdoch University
SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving
Apr 21, 2026Jinda Jia, Jisen Li, Zhongzhu Zhou +8LLM Serving KV Caching
Together AI
Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition
Apr 20, 2026Ziyang LiuLLM Quantization 4-Bit Quantization
Independent Researcher
LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization
Apr 20, 2026Yann Bouquet, Alireza Khodamoradi, Sophie Yáng Shen +2Mixed-Precision Quantization Diffusion Transformer
Research and Advancement Development Team, Advanced Micro Devices, Inc, USA · Computer Vision Laboratory, Ecole Polytechnique F´ed´erale de Lausanne, Lausanne, Switzerland.
DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization
Apr 20, 2026Haokun Lin, Xinle Jia, Haobo Xu +7LLM Quantization 4-Bit Quantization
CASIA · CityU · NJU +3
When W4A4 Breaks Camouflaged Object Detection: Token-Group Dual-Constraint Activation Quantization
Apr 18, 2026Tianqi Li, Wenyu Fang, Xin He +34-Bit Quantization Activation Quantization
VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · Institute for Infocomm Research, A*STAR +2

Jun 7, 2026, cs.DCJ/K move · Enter open · S save

APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

Hong Guo, Nianhui Guo, Weixing Wang, Jona Otholt+2

Hasso Plattner Institute · German University of Digital Science

LLM Quantization GPU Kernel Optimization 4-Bit Quantization LLM Inference Acceleration Neural Network Quantization

Open paper AskPDF arXiv

W4A4 quantization promises full utilization of INT4 Tensor Cores, yet group dequantization overhead on CUDA Cores has driven existing systems to mixed-precision fallbacks. We present the first systematic study of how intra-SM compute balance governs this bottleneck. Through controlled benchmarks across four GPUs from Ampere and Ada architectures, we identify the Tensor Cores to CUDA Cores throughput ratio (

ρ

) as the primary hardware indicator: the W4A4-g128 kernel yields

2.0

2.5\times

speedup on RTX3090 ( $ρ=16$ ) yet degrades to $0.43$ -- $0.47\times$ on A100 ( $ρ=64$ ) in compute-bond scenarios, establishing W4A4 viability as platform-dependent rather than universally infeasible. Guided by this finding, we build \textbf{APEX4}, which co-designs pure INT4 GEMM kernels with $ρ$ -aware granularity adaptation to mitigate the CUDA Cores dequantization bottleneck. APEX4 achieves perplexity within 0.63 of FP16 on LLaMA-2-70B and outperforms W4Ax Atom-g128 by 4.0%--4.4% in zero-shot accuracy. Deployed as a drop-in replacement in unmodified vLLM, it delivers up to $1.66\times$ end-to-end speedup on L40S ( $ρ=8$ ), and $1.78\times$ on RTX3090 (

ρ=16

2.09\times

on A40 (

ρ=16

), while recovering A100 (

ρ=64

) to

1.20

1.40\times

via the mixed-granularity mode. Our code is available at https://github.com/APEX4-W4A4/APEX4-W4A4.

Momentum

Related topics

Latest papers 65