Bit Training

Bit training focuses on reducing the numerical precision used to represent neural network weights and activations, aiming to accelerate training and reduce memory consumption without significant accuracy loss. Current research explores various bit-widths (4-bit, 8-bit, 16-bit), often employing techniques like quantization and specialized arithmetic to maintain performance, with applications to transformer models and large language models (LLMs) being particularly prominent. This approach holds significant promise for making deep learning more efficient and accessible, enabling training of larger models on resource-constrained hardware and facilitating faster model updates.

Papers

November 30, 2023

Continuous 16-bit Training: Accelerating 32-bit Pre-Trained Neural Networks
Juyoung Yun
Neural Network Deep Learning Training Data Pre Trained Model Efficient Deep Continuous Speech Low Bit Bit Training

August 12, 2023

One-bit Flip is All You Need: When Bit-flip Attack Meets Model Training
Jianshuo Dong, Han Qiu, Yiming Li, Tianwei Zhang, Yuanjie Li, Zeqi Lai, Chao Zhang, Shu-Tao Xia
Model Training Malicious Model Bit Flip Attack Bit Flip Flip Attack Bit Training

June 21, 2023

Training Transformers with 4-bit Integers
Haocheng Xi, Changhao Li, Jianfei Chen, Jun Zhu
Neural Network Transformer Training Low Bit Bit Training

May 23, 2023

QLoRA: Efficient Finetuning of Quantized LLMs
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer
Fine Tuning Small Datasets LLM Quantization Finetuning Method Conversational Search Benchmark Bit Training

November 19, 2022

Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Training
Simla Burcu Harma, Ayan Chakraborty, Nicholas Sperry, Babak Falsafi, Martin Jaggi, Yunho Oh
Low Precision Accuracy Improvement DNN Training Bit Training

February 19, 2022

Bit-wise Training of Neural Network Weights
Cristian Ivan
Neural Network Regularization Technique Neural Network Weight Bit Depth Bit Training