Quantization-Aware Distillation

Latest papers

14 papers5 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models
Oct 7, 2026Wenjun Wang, Heng Li, Yanggan Gu +1LLM Quantization Quantization
The Hong Kong Polytechnic University · Sun Yat-sen University · PolyU-Daya Bay Technology and Innovation Research Institute
Progressive Multi-Ancestor Bit-Depth Distillation
Oct 2, 2026Adil Mubashir Chaudhry, Osama Ahmad, Zubair Khalid +1Quantization-Aware Distillation Model Compression
LUMS School of Science and Engineering · University of Massachusetts Amherst, USA
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
Sep 30, 2026Weili Xu, Jisen Li, Yuqing Jian +8LLM Quantization Quantization-Aware Training
Together AI · University of Illinois Urbana-Champaign · The University of Texas at Austin
JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantization
Sep 27, 2026Kaicheng Yang, Kaisen Yang, Chunyu Liu +8Neural Network Quantization Diffusion Model Quantization
Shanghai Jiao Tong University · Tsinghua University · The Hong Kong Polytechnic University
Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
Sep 22, 2026Yuanteng Chen, Zhilei Liu, Peisong Wang +7Low-Bit Quantization LLM Quantization
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +2
Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs
Aug 21, 2026Bakbergen Ryskulov, Iker García-Ferrero, David Montero +5Quantization-Aware Distillation LLM Quantization
Multiverse Computing
SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features
Aug 11, 2026HyeonJun Lee, Hyeonsik Jo, Jinwoo Chung +1Quantization-Aware Training Quantization-Aware Distillation
Kookmin University
BiSCo-LLM: Lookup-Free Binary Spherical Coding for Extreme Low-Bit Large Language Model Compression
Jul 9, 2026Yuantian Shao, Peisong Wang, Zhilei Liu +6LLM Compression Weight-Only Quantization
Nanjing University of Science and Technology, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Huawei, China
SPARK: Spatial Policy-driven Adaptive Reinforcement learning for Knowledge distillation
Jun 13, 2026Mohamed Jismy Aashik Rasool, Shabir Ahmad, Gisong Oh +1Quantization-Aware Training Image Restoration
Department of IT Convergence Engineering, Gachon University, Sujeong-gu, Seongnam-si, Gyeonggi-do 461-701, Republic of Korea
Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation
Jun 4, 2026Fangbo Tu, Junhua Zhao, Chi Liu +4LLM Quantization Quantization-Aware Distillation
PayPal, Inc.
EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation
Apr 10, 2026Shu-Hao Zhang, Le-Tong Huang, Xiang-Sheng Deng +5Quantization-Aware Distillation LLM Quantization
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210063, China · School of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China · Microsoft AI, Beijing 100080, China +1
Quantization with Unified Adaptive Distillation to enable multi-LoRA based one-for-all Generative Vision Models on edge
Mar 31, 2026Sowmya Vajrala, Aakash Parmar, Prasanna R +4Quantization-Aware Distillation On-Device Inference
Samsung Research Institute Bangalore, India
Decoder-Free Distillation for Quantized Image Restoration
Mar 10, 2026S. M. A. Sharif, Abdur Rehman, Seongwan Kim +1Image Restoration Quantization-Aware Distillation
Opt-AI Inc., Seoul, South Korea
From Fewer Samples to Fewer Bits: Reframing Dataset Distillation as Joint Optimization of Precision and Compactness
Mar 2, 2026My H. Dinh, Aditya Sant, Akshay Malhotra +2Quantization-Aware Distillation Dataset Distillation
InterDigital Communications, Inc.

Oct 7, 2026J/K move · Enter open · S save

OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models

Wenjun Wang, Heng Li, Yanggan Gu, Hongxia Yang

The Hong Kong Polytechnic University · Sun Yat-sen University · PolyU-Daya Bay Technology and Innovation Research Institute

LLM Quantization Quantization Quantization-Aware Training Quantization-Aware Distillation On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

Quantization-aware training (QAT) can recover much of the accuracy lost when large language models are compressed below four bits. Existing re- covery stages, however, are commonly optimized on fixed completions or teacher-generated answers, whereas the deployed quantized model condi- tions on prefixes generated by itself. Quantization errors can therefore move the model into states that are absent from offline recovery data. We introduce OnlineQAT, a two-stage framework that first obtains a usable low-bit initialization through block-wise QAT and then performs on-policy distillation (OPD) on student-generated responses. At each visited pre- fix, a frozen full-precision teacher provides a sampled reverse-KL training signal. On Qwen3-1.7B, OnlineQAT obtains the best average among the compared quantized methods: 57.28 at W3A16 and 32.52 at W2A16, im- proving over ReasoningQAT by 2.90 and 0.44 points, respectively. The results suggest that student-visited states provide a useful recovery signal beyond fixed-completion training, particularly at three bits.