Weight-Only Quantization

Follow this topic Ask about this topic

Weight-only quantization studies how to represent a model’s weights with fewer bits while preserving its behavior, especially when deploying large language models on memory-limited hardware. Methods range from rounding and group-wise post-training quantization to activation- or curvature-aware error correction, learned codebooks, rotations, and quantization-aware training; mixed precision can protect sensitive layers or weights. Work is evaluated using measures such as language-model perplexity, zero-shot and reasoning accuracy, generation quality, memory use, and throughput on target hardware. The main tension is that very low bit rates save substantial storage but can sharply damage capabilities, while better accuracy may require calibration, extra metadata, or kernels whose real speed depends on the hardware and workload.

Latest papers

76 papers18 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

Every Weight Is a Path of Shared Discrete Decisions
Oct 22, 2026Ammar Ali, Stamatis LefkimmiatisNeural Network Compression Neural Network Hardware Acceleration
LEAD Lab, ITMO University
ProQuant: Preserving Expert Quality over Quantity in Ultra-Low-Bit Quantization of Mixture-of-Experts Models
Oct 22, 2026Jiaqi Chen, Xianglong Yan, Chengzhu Bao +2Mixture-of-Experts Quantization Post-Training Quantization
Shanghai Jiao Tong University
FIGFLEX: A Mixed-Precision FP-INT GEMM Accelerator Co-Designed with Low-Bit LLM Quantization
Oct 21, 2026Faraz Tahmasebi, Haocheng Xu, Andrew Trauger +4LLM Inference Acceleration LLM Quantization
University of California, Irvine, Irvine, CA, USA
AGQ: Action-Guided Ultra-Low-Bit Quantization for Vision-Language-Action Models
Oct 21, 2026Xuhong Qiu, Zhuoyu Wu, Qiawei Zheng +3VLM Quantization Efficient VLA Model Inference
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Information Technology, Monash University, Subang Jaya, Malaysia · Department of Electronic & Computer Engineering, Hong Kong University of Science & Technology, Hong Kong, China +1
RoleQ: Role-Aligned Scales for Converting Language Models to Ternary
Oct 15, 2026Johannes Knittel, Alec Helbling, Hanspeter PfisterLLM Quantization Neural Network Quantization
Harvard University · Georgia Tech
Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference
Oct 13, 2026Petar BastaLLM Inference Efficiency LLM Quantization
University of Waterloo Computer Science Waterloo, Ontario, Canada
FidelityQ: Preserving Predictive Fidelity in Ultra-Low-Bit Language Model Quantization
Oct 10, 2026Yujie Chen, Weiyan Zheng, Yiwu Yao +7Low-Bit Quantization Neural Network Quantization
The Hong Kong Polytechnic University · Huawei Technologies Ltd. · The University of Hong Kong +2
SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability
Oct 10, 2026Xianglong Yan, Zenan Wu, Kewei Zhang +2LLM Quantization Weight-Only Quantization
Shanghai Jiao Tong University
When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization
Oct 8, 2026Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu +6LLM Quantization Post-Training Quantization
University of Science and Technology of China · University College London · Shanghai University of Finance and Economics +5
CurveTQ: Rotation-Free Trellis Quantization of LLM Weights via Curvature-Weighted Search
Oct 6, 2026Guanhua Ding, Zi Wang, Ruichao Li +1LLM Quantization Neural Network Quantization
Black Sesame Technologies
StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs
Oct 5, 2026Zhe Wei, Mengqi Guo, Yuan Yuan +3Mixed-Precision Quantization LLM Quantization
Huawei Technologies Ltd.
BARQ: Balanced Codebook Refinement for Low-Bit LLM Quantization
Oct 3, 2026Chenhang Cui, Xu Xie, Linrui Xu +4LLM Quantization Low-Bit Quantization
National University of Singapore · Independent Researcher · Central South University
JARQ: Joint Alternating Refinement for Quantization
Sep 29, 2026Xinyu Wang, Sicheng Lyu, Xiao-Wen ChangLLM Quantization Weight-Only Quantization
McGill University
ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights
Sep 29, 2026Jonathan Mei, Sang Hyub Kim, Oliver Knitter +2LLM Quantization Weight-Only Quantization
Applications R&D IonQ College Park, MD 20740, USA
Security-Enhanced Seed-Based Weight Quantization for Large Language Models
Sep 29, 2026Qiuyu Ren, Sudipta Paria, Aritra Dasgupta +1LLM Quantization LLM Compression
Department of Electrical and Computer Engineering (ECE) University of Florida Gainesville, FL 32611, USA
Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter
Sep 28, 2026Pier-Jean MalandrinoLLM Quantization LLM Inference Acceleration
Scub, Bordeaux, France
Fiona: Accelerating FHE Inference with Packing-Aware Ternary Weights
Sep 28, 2026Yiteng Peng, Zhibo Liu, Dongwei Xiao +1Mixed-Precision Quantization Efficient Neural Network Inference
The Hong Kong University of Science and Technology · State Key Laboratory of Novel Software Technology, Nanjing University
EntroPack: Fast and Accurate Entropy-Coded Weight Compression at Arbitrary Bitrates
Sep 28, 2026Hong Zhang, Zhongjie Duan, Yingda ChenEfficient Neural Network Inference Weight-Only Quantization
Alibaba Inc.
PulseQuant: Propagation-Guided Subspace Correction for 4-Bit Video Diffusion Transformers
Sep 27, 2026Yutong Wang, Xingtong Ge, Enhuai Liu +4Video Diffusion Models Diffusion Transformer
The University of Sydney · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +1
Same Bit Width, Different Outcomes: Post-Training Quantization of Text-to-Speech Across Architectures
Sep 24, 2026Se Un Park, Yutae Kim, Junyoung ParkTTS Evaluation Weight-Only Quantization
UX Factory, Inc.
Beyond Scalar Sensitivity: Activation-Aware Mixed-Precision LLM Quantization with Cross-Layer Refinement
Sep 22, 2026Akihiro Yoshida, Yuma IchikawaMixed-Precision Quantization LLM Quantization
Fujitsu Limited, Institute of Science Tokyo · Fujitsu Limited, RIKEN Center for AIP
SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models
Sep 21, 2026Kewei Zhang, Zheng Chen, Haotong Qin +1VLM Quantization Efficient VLM Inference
Shanghai Jiao Tong University · The Hong Kong Polytechnic University
ShatterQuant: Breaking Uniform Precision with Block-Wise Mixed-Precision on a Systolic Transformer Hardware Accelerator
Sep 20, 2026Mikolaj Walczak, Edward Humes, Chao Fang +2Mixed-Precision Quantization AI Accelerator Inference
Johns Hopkins University Baltimore, Maryland, USA
Global Ranks Survive, Selected Heads Shift: BOS-Sink Topology under 4-bit Weight-Only Quantization
Sep 20, 2026Kuanlin Chen, Chen-Wei Kuo, Cheng-En OuAttention Head Analysis 4-Bit Quantization
Independent Researcher Taoyuan, Taiwan · National Tsing Hua University Hsinchu, Taiwan
Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families
Sep 14, 2026Hyojung HanText Embedding Evaluation Weight-Only Quantization
ThakiCloud
Structured Transforms for Low-Overhead Quantization of Language Models
Sep 10, 2026Daria Cherniuk, Alexander Rudikov, Boris Kashin +1LLM Quantization Weight-Only Quantization
Institute of Numerical Mathematics · Steklov Mathematical Institute
Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution
Sep 8, 2026Anirudh Malik, M Sparsh Mehra, Poojith DevanLLM Quantization Ternary Quantization
OneBit AI
Quantization Amplifies Determinism, Not Bias: Scale-Dependent Behavioral Effects of Serving-Time Weight Compression
Sep 7, 2026Dachi KurtskhaliaLLM Quantization Weight-Only Quantization
Impel
Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs
Sep 6, 2026Saurav Bhandari, Benjamin WadeActivation Steering LLM Quantization
Department of Electronics and Computer Engineering Institute of Engineering, Thapathali Campus Tribhuvan University, Kathmandu, Nepal · Independent Researcher
Unfolding the Leech Lattice: Fused Multi-Shell Decoding and VRAM Layouts for 2-Bit LLM Weights
Sep 2, 2026Pier-Jean MalandrinoLLM Inference Vector Quantization
Scub

Oct 22, 2026J/K move · Enter open · S save

Every Weight Is a Path of Shared Discrete Decisions

Ammar Ali, Stamatis Lefkimmiatis

LEAD Lab, ITMO University

Neural Network Compression Neural Network Hardware Acceleration Neural Network Quantization Weight-Only Quantization

Open paper AskPDF arXiv

We introduce TWIG (Tree-structured Weights with Inherited Gates), a representation in which every weight of a neural network is the end of a path of small discrete decisions: a sign, a shift by a power of two, a zero gate, an index into a shared table. Weights that belong together share the upper part of their paths, and each decision is inherited by every weight below it, so it is stored once and applied once. This simple change has three consequences. Any trained network can be written as a TWIG, and when the decisions are powers of two its weight layers become circuits made almost entirely of wires and adders, suited to chips whose weights are fixed at fabrication. Pruning, quantization, low-rank factorization, codebooks and weight sharing become edits of one object, so a single optimizer can choose among all of them for a whole network. And choosing the decisions is the ground-state problem of a higher-order Boltzmann machine whose interactions follow the paths of the tree, the problem class that annealers and quantum optimizers are built to solve. Today these questions are handled apart: each compression method brings its own format and solver, a circuit with fixed weights needs a dedicated constant multiplier for every weight, and discrete solvers reach only narrow subproblems such as rounding. Weight formats have long been moving toward shared decisions, through group scales, shared exponents, microexponents and sparse weights; TWIG takes this trend to its end, and every common format is a shallow TWIG at no extra cost. On five weight matrices of a large language model, a flat tree whose signed power-of-two terms are chosen per weight for circuit cost, with no group scales, needs 9--15% fewer additions than GPTQ at equal layer error at 3- and 4-bit accuracy on typical matrices, and 20--55% fewer on matrices with strong outliers.