Int8 Training

Int8 training aims to accelerate and improve the efficiency of training large neural networks by using 8-bit floating-point numbers instead of higher-precision formats like FP16 or FP32. Current research focuses on overcoming instabilities inherent in low-precision training, particularly for large language models (LLMs) and vision-language models, often employing techniques like modified activation functions and optimized AdamW variants. These advancements significantly reduce training time and memory requirements, leading to substantial improvements in the scalability and accessibility of training massive models for various applications, including natural language processing and computer vision.

Papers

October 25, 2024

COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training
Haocheng Xi, Han Cai, Ligeng Zhu, Yao Lu, Kurt Keutzer, Jianfei Chen, Song Han
Model Training Targeted Activation Penalty FP8 Format Int8 Training

September 19, 2024

Scaling FP8 training to trillion-token LLMs
Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry
Large Language Model Weight Re Mapping Synthetic Outlier Floating Point Quantization Int8 Training

July 2, 2024

Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point
Bokun Wang, Axel Berg, Durmus Alp Emre Acar, Chuteng Zhou
Federated Learning Timely Communication Exceptional Point Device Training FP8 Format Int8 Training

March 19, 2024

Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization
Haocheng Xi, Yuxiang Chen, Kang Zhao, Kaijun Zheng, Jianfei Chen, Jun Zhu
High Efficiency Pre Trained Transformer Dataflow Architecture Training Throughput Efficient Dequantization Block Quantization Int8 Training

April 25, 2023

Stable and low-precision training for large-scale vision-language models
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, Ludwig Schmidt
Low Precision Training Int8 Training

Int8 Training

Papers

COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training

Scaling FP8 training to trillion-token LLMs

Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point

Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization

Stable and low-precision training for large-scale vision-language models