Mixed Precision Quantization

Mixed-precision quantization aims to improve the efficiency of deep neural networks by assigning different numerical precisions (bit-widths) to various layers or components, balancing accuracy and resource consumption. Current research focuses on developing algorithms that automatically determine optimal bit-width allocations for various architectures, including transformers (Vision Transformers, and Large Language Models) and convolutional neural networks, often incorporating techniques like quantization-aware training and hardware-aware optimization. This optimization technique is crucial for deploying large models on resource-constrained devices like embedded systems and mobile platforms, impacting both the accessibility and energy efficiency of AI applications.

Papers

November 9, 2024

Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques
Jahid Hasan
General Analysis Quantization Operator Quantization Aware Training Mixed Precision Quantization Quantization Technique

October 31, 2024

ARQ: A Mixed-Precision Quantization Framework for Accurate and Certifiably Robust DNNs
Yuchen Yang, Shubham Ugare, Yifan Zhao, Gagandeep Singh, Sasa Misailovic
Robust Deep Mixed Precision Quantization State of the Art Quantization

October 16, 2024

Channel-Wise Mixed-Precision Quantization for Large Language Models
Zihan Chen, Bike Xie, Jundong Li, Cong Shen
Mixed Precision Quantization Channel Wise Weight Only Quantization Accurate Quantization Non Uniform Quantization

October 9, 2024

Scaling Laws for Mixed quantization in Large Language Models
Zeyu Cao, Cheng Zhang, Pedro Gimenes, Jianqiao Lu, Jianyi Cheng, Yiren Zhao
Scaling Law Mixed Precision Quantization Mixed Quantization Low Precision Quantization Quantization Granularity

October 8, 2024

MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More
Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, Xiaojuan Qi
Large Language Model Language Model Mixture of Expert Mixed Precision Quantization Dynamic Pruning Compressor Based Machine

October 4, 2024

Resource-aware Mixed-precision Quantization for Enhancing Deployability of Transformers for Time-series Forecasting on Embedded FPGAs
Tianheng Ling, Chao Qian, Gregor Schiele
Transformer Megatron Decepticons Time Series Forecasting Field Programmable Gate Array Mixed Precision Quantization Uncertainty Aware Deployment FPGA Resource Heterogeneous Quantization

July 25, 2024

Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers
Zhengang Li, Alec Lu, Yanyue Xie, Zhenglun Kong, Mengshu Sun, Hao Tang, Zhong Jia Xue, Peiyan Dong, Caiwen Ding, Yanzhi Wang, Xue Lin, Zhenman Fang
Convolutional Neural Network Vision Transformer Mixed Precision Quantization ViT Model

July 19, 2024

Mixed-precision Neural Networks on RISC-V Cores: ISA extensions for Multi-Pumped Soft SIMD Operations
Giorgos Armeniakos, Alexis Maras, Sotirios Xydis, Dimitrios Soudris
Mixed Precision Quantization Mixed Precision Risc V

July 1, 2024

Joint Pruning and Channel-wise Mixed-Precision Quantization for Efficient Deep Neural Networks
Beatrice Alessandra Motetti, Matteo Risso, Alessio Burrello, Enrico Macii, Massimo Poncino, Daniele Jahier Pagliari
Deep Neural Network Edge Computing Pareto Optimal Mixed Precision Quantization Efficient Deep Mixed Precision

June 13, 2024

ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models
Jing Liu, Ruihao Gong, Mingyang Zhang, Yefei He, Jianfei Cai, Bohan Zhuang
Mixed Precision Quantization Switch List Lossless Performance Pre Trained Model Weight

June 10, 2024

Efficient Neural Compression with Inference-time Decoding
C. Metz, O. Bichler, A. Dupret
Inference Time Mixed Precision Quantization Neural Compression Fixed Point Neural Network Quantization Decoder Architecture

June 8, 2024

Towards Lightweight Speaker Verification via Adaptive Neural Network Quantization
Bei Liu, Haoyu Wang, Yanmin Qian
Mixed Precision Quantization Neural Network Quantization Adaptive Quantization Precision Quantization

May 30, 2024

P$^2$-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformer
Huihong Shi, Xin Cheng, Wendong Mao, Zhongfeng Wang
Vision Transformer Quantization Operator Low Rate ACCELERATION Mixed Precision Quantization ViT Model Model Re Quantization

May 28, 2024

MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization
Tianchen Zhao, Xuefei Ning, Tongcheng Fang, Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, Yu Wang
Quantization Operator Mixed Precision Quantization

May 23, 2024

SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models
Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, Xiaojuan Qi
Mixed Precision Quantization LLM Model LLM Quantization Bit Allocation

May 1, 2024

Gradient-based Automatic Mixed Precision Quantization for Neural Networks On-Chip
Chang Sun, Thea K. Årrestad, Vladimir Loncar, Jennifer Ngadiuba, Maria Spiropulu
Multiplier Free Quantization Quantization Aware Training Mixed Precision Quantization Uniform Quantization Chip Learning Quantization Granularity

April 22, 2024

AdaQAT: Adaptive Bit-Width Quantization-Aware Training
Cédric Gernigon, Silviu-Ioan Filip, Olivier Sentieys, Clément Coggiola, Mickael Bruno
Deep Neural Network Large Scale Quantization Aware Training Mixed Precision Quantization

March 2, 2024

LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
Juntao Zhao, Borui Wan, Yanghua Peng, Haibin Lin, Chuan Wu
Medical LLM Large Scale Language Model Mixed Precision Quantization Inference Workload Adaptive Quantization Heterogeneous GPU Heterogeneous Cluster Model Partition

February 28, 2024

No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, Dongsoo Lee
K TOKEN Mixed Precision Quantization Key Value Cache KV Cache Key Value Cache Compression

February 27, 2024

Adaptive quantization with mixed-precision based on low-cost proxy
Junzhe Chen, Qiao Yang, Senmao Tian, Shunli Zhang
Mixed Precision Quantization Model Quantization Zero Cost Proxy Adaptive Quantization Accurate Quantization Quantization Parameter

Mixed Precision Quantization

Papers

Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques

ARQ: A Mixed-Precision Quantization Framework for Accurate and Certifiably Robust DNNs

Channel-Wise Mixed-Precision Quantization for Large Language Models

Scaling Laws for Mixed quantization in Large Language Models

MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More

Resource-aware Mixed-precision Quantization for Enhancing Deployability of Transformers for Time-series Forecasting on Embedded FPGAs

Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers

Mixed-precision Neural Networks on RISC-V Cores: ISA extensions for Multi-Pumped Soft SIMD Operations

Joint Pruning and Channel-wise Mixed-Precision Quantization for Efficient Deep Neural Networks

ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models

Efficient Neural Compression with Inference-time Decoding

Towards Lightweight Speaker Verification via Adaptive Neural Network Quantization

P$^2$-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformer

MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization

SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

Gradient-based Automatic Mixed Precision Quantization for Neural Networks On-Chip

AdaQAT: Adaptive Bit-Width Quantization-Aware Training

LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization

No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization

Adaptive quantization with mixed-precision based on low-cost proxy