FP8 Format

FP8, an 8-bit floating-point format, aims to accelerate deep learning training and inference by significantly reducing computational costs and memory usage compared to higher-precision formats like FP16 and FP32. Current research focuses on developing robust FP8 training methods for large language models (LLMs) and other deep learning architectures, including convolutional neural networks (CNNs) and transformers, investigating optimal encoding schemes (e.g., varying exponent and mantissa bit allocations), and comparing FP8's performance against INT8 for inference. Successful implementation of FP8 could dramatically reduce the computational burden of training and deploying large-scale AI models, impacting both research and practical applications.

Papers

November 10, 2024

Balancing Speed and Stability: The Trade-offs of FP8 vs. BF16 Training in LLMs
Kazuki Fujii, Taishi Nakamura, Rio Yokota
Large Language Model Natural Language Model Performance Large Scale Model Balance Control FP8 Format

October 25, 2024

COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training
Haocheng Xi, Han Cai, Ligeng Zhu, Yao Lu, Kurt Keutzer, Jianfei Chen, Song Han
Model Training Targeted Activation Penalty FP8 Format Int8 Training

September 25, 2024

Ascend HiFloat8 Format for Deep Learning
Yuanyong Luo, Zhongxing Zhang, Richard Wu, Hu Liu, Ying Jin, Kai Zheng, Minmin Wang, Zhanying He, Guipeng Hu, Luyao Chen, Tianchi Hu, Junsong Wang, Minqi Chen, Mikhaylov Dmitry, Korviakov Vladimir, Bobrin Maxim, Yuhao Hu, Guanfu Chen, Zeyi Huang
Deep Learning Data Encoding Novel Encoding FP8 Format Floating Point Format

July 2, 2024

Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point
Bokun Wang, Axel Berg, Durmus Alp Emre Acar, Chuteng Zhou
Federated Learning Timely Communication Exceptional Point Device Training FP8 Format Int8 Training

May 29, 2024

To FP8 and Back Again: Quantifying the Effects of Reducing Precision on LLM Training Stability
Joonhyung Lee, Jeongin Bae, Byeongwook Kim, Se Jung Kwon, Dongsoo Lee
Large Language Model Mixed Effect Low Precision Floating Point Low Precision Training FP8 Format

October 27, 2023

FP8-LM: Training FP8 Large Language Models
Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, Ruihang Li, Miaosen Zhang, Chen Li, Jia Ning, Ruizhe Wang, Zheng Zhang, Shuguang Liu, Joe Chau, Han Hu, Peng Cheng
Instruction Tuning Large Language Model Training Low Precision Training Mixed Precision Training FP8 Format

September 29, 2023

Training and inference of large language models using 8-bit floating point
Sergio P. Perez, Yan Zhang, James Briggs, Charlie Blake, Josh Levy-Kramer, Paul Balanca, Carlo Luschi, Stephen Barlow, Andrew William Fitzgibbon
Large Language Model Training Data Scientific Inference Exceptional Point Large Deep Learning Model FP8 Format

September 26, 2023

Efficient Post-training Quantization with FP8 Formats
Haihao Shen, Naveen Mellempudi, Xin He, Qun Gao, Chang Wang, Mengni Wang
Post Training Quantization Neural Compression FP8 Format

March 31, 2023

FP8 versus INT8 for efficient deep learning inference
Mart van Baalen, Andrey Kuzmin, Suparna S Nair, Yuwei Ren, Eric Mahurin, Chirag Patel, Sundar Subramanian, Sanghyuk Lee, Markus Nagel, Joseph Soriaga, Tijmen Blankevoort
Deep Learning Device Inference Personality Classification Deep Learning Inference Artificial Intelligence Inference Bit Weight FP8 Format

September 12, 2022

FP8 Formats for Deep Learning
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, Hao Wu
Deep Learning Post Training Quantization Binary Code FP8 Format INT4 Quantization Modern Neural Network Architecture