Training Deep

Training deep neural networks efficiently and effectively remains a central challenge in machine learning. Current research focuses on improving training algorithms (e.g., exploring second-order methods and adaptive gradient normalization), optimizing model architectures (e.g., reversible architectures and sparse mixtures of experts), and reducing computational costs (e.g., through gradient sampling, model compression, and efficient distributed training). These advancements aim to enhance model performance, reduce energy consumption, and enable training on larger datasets or resource-constrained devices, impacting various applications from medical image analysis to financial modeling.

Papers

November 30, 2021

AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop
Yunfei Teng, Jing Wang, Anna Choromanska
Stochastic Gradient Descent Learning Rate Training Deep Modern Deep Learning Saturation Effect

November 20, 2021

Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters
Adarsh Kumar, Kausik Subramanian, Shivaram Venkataraman, Aditya Akella
Back Propagation Training Deep Deep Learning Workload Computing Cluster Structured Neural Network

November 12, 2021

Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment
Matthew J. Filipovich, Zhimu Guo, Mohammed Al-Qadasi, Bicky A. Marquez, Hugh D. Morison, Volker J. Sorger, Paul R. Prucnal, Sudip Shekhar, Bhavin J. Shastri
Training Deep Chip Learning Silicon Photonics Feedback Alignment Photonic Circuit Fast Neural Network Photonic Processor

Training Deep

Papers

AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop

Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters

Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment