Ternary Quantization

Ternary quantization is a model compression technique that reduces the memory footprint and computational cost of deep neural networks by representing weights as only -1, 0, and 1. Current research focuses on improving the accuracy of ternary quantized models, particularly for vision transformers (ViTs) and large language models (LLMs), through techniques like optimized quantization algorithms (e.g., those leveraging residual error expansion or hyperspherical learning) and refined training methods (e.g., quantization-aware training). This research is significant because it enables the deployment of powerful deep learning models on resource-constrained devices, broadening the accessibility and applicability of AI across various domains.

Papers

October 8, 2024

Accelerating Error Correction Code Transformers
Matan Levy, Yoni Choukroun, Lior Wolf
Low Precision Quantization Ternary Quantization Transformer Based Decoder

June 26, 2024

ViT-1.58b: Mobile Vision Transformers in the 1-bit Era
Zhengqing Yuan, Rong Zhou, Hongyi Wang, Lifang He, Yanfang Ye, Lichao Sun
Vision Transformer New Era ImageNet 1k Ternary Quantization Mobile Vision Transformer

November 27, 2023

PIPE : Parallelized Inference Through Post-Training Quantization Ensembling of Residual Expansions
Edouard Yvinec, Arnaud Dapogny, Kevin Bailly
Deep Neural Network Scientific Inference Multiplier Free Quantization High Performing 1D ConvNet Residual Model Product Quantization Software Evolution Ternary Quantization

August 15, 2023

Ternary Singular Value Decomposition as a Better Parameterized Form in Linear Mapping
Boyu Chen, Hanxuan Chen, Jiao He, Fengyu Sun, Shangling Jui
Parameter Efficient Singular Value Decomposition Ternary Quantization Linear Mapping

June 30, 2023

Designing strong baselines for ternary neural network quantization through support and mass equalization
Edouard Yvinec, Arnaud Dapogny, Kevin Bailly
Customer Service Strong Baseline Quantization Aware Training Quantization Error Ternary Neural Network Ternary Quantization Equalization Great

March 2, 2023

Ternary Quantization: A Survey
Dan Liu, Xue Liu
Timely Survey Model Quantization Low Bit Quantization Quantization Step Ternary Quantization

December 24, 2022

October 31, 2022

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh
Transformer Megatron Decepticons Generative Question Post Training Quantization Pre Trained Transformer Real Time Inference Zero Shot Quantization Ternary Quantization

March 28, 2022

REx: Data-Free Residual Quantization Error Expansion
Edouard Yvinec, Arnaud Dapgony, Matthieu Cord, Kevin Bailly
Deep Neural Network Multiplier Free Quantization High Performing 1D ConvNet Product Quantization Residual Quantization Ternary Quantization