Precision Transformer

Precision Transformer research focuses on optimizing the efficiency and performance of transformer-based models, primarily large language models (LLMs), by reducing their computational and memory requirements. Current efforts concentrate on techniques like post-training quantization (PTQ), including methods employing learned rotations to minimize quantization errors, and the exploration of extremely low-precision (e.g., binary, ternary, 1-bit) architectures. These advancements aim to improve the scalability and accessibility of LLMs by significantly reducing energy consumption and memory footprint while maintaining or even improving accuracy on various downstream tasks.

Papers

May 27, 2024

Mechanistic Interpretability of Binary and Ternary Transformers
Jason Li
Mechanistic Interpretability Binary Code Full Precision Precision Transformer Ternary Neural Network

May 26, 2024

SpinQuant: LLM quantization with learned rotations
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, Tijmen Blankevoort
Post Training Quantization Quantization Error LLM Quantization Precision Transformer

March 14, 2024

Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models
Akhil Kedia, Mohd Abbas Zaidi, Sushil Khyalia, Jungho Jung, Harshith Goka, Haejun Lee
Language Model Transformer Megatron Decepticons Transformer Model Deep Model Propagation Environment Precision Transformer

October 17, 2023

BitNet: Scaling 1-bit Transformers for Large Language Models
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
Large Language Model Transformer Megatron Decepticons Transformer Architecture Larger Language Model Precision Transformer

August 9, 2023

Sparse Binary Transformers for Multivariate Time Series Modeling
Matt Gorbett, Hossein Shirazi, Indrakshi Ray
Neural Network Deep Learning Time Series Multivariate Time Series Lightweight Model Sparse Transformer Precision Transformer

August 6, 2023

Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuits
Lena Strobl
Transformer Model Precision Transformer Threshold Circuit

February 23, 2023

Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers
Minsoo Kim, Kyuhong Shim, Seongmin Park, Wonyong Sung, Jungwook Choi
Early Stage Convergence Pre Trained Transformer Quantization Aware Training Quantization Technique Precision Transformer

October 6, 2022

A Logic for Expressing Log-Precision Transformers
William Merrill, Ashish Sabharwal
Transformer Based Language Model Logic Reasoning Precision Transformer Transformer Variant

July 2, 2022

The Parallelism Tradeoff: Limitations of Log-Precision Transformers
William Merrill, Ashish Sabharwal
Transformer Based Transformer Architecture Fundamental Limitation Trade Offs Computational Power Precision Transformer Threshold Circuit

March 23, 2022

Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models
Ali Hadi Zadeh, Mostafa Mahmoud, Ameer Abdelhadi, Andreas Moshovos
Transformer Model Shelf Model Precision Transformer

November 29, 2021

Mixed Precision of Quantization of Transformer Language Models for Speech Recognition
Junhao Xu, Shoukang Hu, Jianwei Yu, Xunying Liu, Helen Meng
Speech Recognition Quantization Operator Transformer Language Model Quantized Neural Network Precision Transformer Quantized Transformer