Per Tensor

Per-tensor quantization aims to improve the efficiency of large language models (LLMs) by representing model weights and activations using fewer bits per tensor, thereby reducing memory footprint and accelerating inference. Current research focuses on mitigating the accuracy loss associated with this compression, employing techniques like low-rank matrix decompositions, outlier detection and pre-processing to optimize quantization schemes (e.g., per-tensor static quantization) and achieve performance comparable to or exceeding dynamic quantization methods. These advancements are significant for deploying LLMs on resource-constrained devices and accelerating their practical applications, particularly in areas like natural language processing and reinforcement learning.

Papers

October 17, 2024

Quamba: A Post-Training Quantization Recipe for Selective State Space Models
Hung-Yueh Chiang, Chi-Chih Chang, Natalia Frumkin, Kai-Chiang Wu, Diana Marculescu
Large Language Model Full Model Long Sequence Selective State Space Accurate Quantization Per Tensor

October 7, 2024

PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs
Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, Ping Luo
Large Language Model Dynamic Quantization Per Tensor

July 16, 2024

LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices
Jung Hyun Lee, Jeonghoon Kim, June Yong Yang, Se Jung Kwon, Eunho Yang, Kang Min Yoo, Dongsoo Lee
Quantization Operator Post Training Quantization Weight Matrix Activation Quantization Low Rank Quantization Per Tensor

June 27, 2024

OutlierTune: Efficient Channel-Wise Quantization for Large Language Models
Jinguang Wang, Yuexi Yin, Haifeng Sun, Qi Qi, Jingyu Wang, Zirui Zhuang, Tingting Yang, Jianxin Liao
Large Language Model Post Training Quantization Activation Quantization Outlier Sample Per Tensor

June 17, 2024

Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization
Seungwoo Son, Wonpyo Park, Woohyun Han, Kyuyeun Kim, Jaeho Lee
Large Language Model Activation Quantization LLM Quantization Per Tensor Attention Sink

February 28, 2024

FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization
Yi Zhang, Fei Yang, Shuang Peng, Fangyu Wang, Aimin Pan
Large Language Model Fast Inference Low Bit Quantization Linear Layer Fine Grained Quantization Per Tensor

April 30, 2022

TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learning
Konstantin Sozykin, Andrei Chertkov, Roman Schutski, Anh-Huy Phan, Andrzej Cichocki, Ivan Oseledets
Reinforcement Learning Application Proficiency Optimization Purpose Tensor Train Evolutionary Method Per Tensor

Per Tensor

Papers

Quamba: A Post-Training Quantization Recipe for Selective State Space Models

PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs

LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices

OutlierTune: Efficient Channel-Wise Quantization for Large Language Models

Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization

FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization

TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learning