Sparse Backpropagation

Sparse backpropagation aims to accelerate and reduce the memory footprint of training deep neural networks by selectively updating only a subset of model parameters during backpropagation. Current research focuses on developing dynamic sparsity algorithms, such as TinyProp and its variants, and applying sparse backpropagation to various architectures, including Mixture-of-Experts (MoE) models and recurrent neural networks. These advancements are particularly impactful for resource-constrained environments like edge devices and embedded systems, enabling on-device learning and fine-tuning of large models while maintaining accuracy. The resulting efficiency gains are significant for both training speed and energy consumption.