Quantized Transformer

Quantized transformers aim to reduce the computational cost and memory footprint of transformer models by representing their weights and activations using fewer bits, thereby enabling deployment on resource-constrained devices. Current research focuses on optimizing quantization techniques, including integer and floating-point quantization, exploring different bit-widths (e.g., 4-bit, 6-bit, 8-bit), and developing novel architectures and algorithms to mitigate information loss during quantization. This work is significant because it addresses the scalability challenges of large transformer models, paving the way for wider adoption in applications like embedded systems, edge AI, and mobile devices.

Papers

July 6, 2024

Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT
Tianheng Ling, Chao Qian, Gregor Schiele
Transformer Model Time Series Forecasting Hardware Accelerator Embedded System Device Inference Integer Quantization Quantized Transformer

March 31, 2024

Transformer based Pluralistic Image Completion with Reduced Information Loss
Qiankun Liu, Yuqi Jiang, Zhentao Tan, Dongdong Chen, Ying Fu, Qi Chu, Gang Hua, Nenghai Yu
Transformer Based Image Fidelity Image Completion Information Loss Quantized Transformer Input Quantization

October 25, 2023

LLM-FP4: 4-Bit Floating-Point Quantized Transformers
Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, Kwang-Ting Cheng
Post Training Quantization Activation Quantization Integer Quantization Quantized Transformer Floating Point Quantization

July 7, 2023

ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers
Gamze İslamoğlu, Moritz Scherer, Gianna Paulin, Tim Fischer, Victor J. B. Jung, Angelo Garofalo, Luca Benini
Transformer Model Softmax Function Transformer Network Hardware Acceleration Accelerator Architecture Transformer Accelerator Quantized Transformer

June 22, 2023

Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort
Language Model Vision Transformer Transformer Model Attention Head Quantized Transformer

May 2, 2023

AQ-GT: a Temporally Aligned and Quantized GRU-Transformer for Co-Speech Gesture Synthesis
Hendric Voß, Stefan Kopp
Co Speech Gesture Gesture Sequence Body Gesture Speech Driven Gesture Quantized Transformer

April 8, 2023

SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers
Alberto Marchisio, Davide Dura, Maurizio Capra, Maurizio Martina, Guido Masera, Muhammad Shafique
Quantized Transformer

September 27, 2022

Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, Xianglong Liu
Language Model Transformer Architecture Continuum Limit Outlier Removal Quantized Transformer

September 25, 2022

SpeedLimit: Neural Architecture Search for Quantized Transformer Models
Yuji Chai, Luke Bailey, Yunho Jin, Matthew Karle, Glenn G. Ko, David Brooks, Gu-Yeon Wei, H. T. Kung
Neural Architecture Search Transformer Model Quantized Transformer Variable Speed Limit

May 10, 2022

Reduce Information Loss in Transformers for Pluralistic Image Inpainting
Qiankun Liu, Zhentao Tan, Dongdong Chen, Qi Chu, Xiyang Dai, Yinpeng Chen, Mengchen Liu, Lu Yuan, Nenghai Yu
Transformer Megatron Decepticons Image Fidelity Information Loss Pluralistic Alignment Q Transformer Quantized Transformer

November 29, 2021

Mixed Precision of Quantization of Transformer Language Models for Speech Recognition
Junhao Xu, Shoukang Hu, Jianwei Yu, Xunying Liu, Helen Meng
Speech Recognition Quantization Operator Transformer Language Model Quantized Neural Network Quantized Transformer Precision Transformer