Deep Neural Network Training

Deep neural network (DNN) training aims to optimize DNN models for accurate and efficient performance across diverse applications. Current research emphasizes improving training efficiency through techniques like gradient sparsification, adaptive gradient prediction, and optimized scheduling for distributed training across multiple devices, including microcontrollers and specialized hardware. These advancements address challenges such as communication bottlenecks in federated learning, resource constraints on edge devices, and the need for robust and reliable models in fault-prone environments. Ultimately, efficient and effective DNN training is crucial for advancing various fields, from industrial automation and healthcare to scientific computing and beyond.

Papers

September 24, 2024

A Multi-Level Approach for Class Imbalance Problem in Federated Learning for Remote Industry 4.0 Applications
Razin Farhan Hussain, Mohsen Amini Salehi
Deep Neural Network Financial Application Class Imbalance Deep Neural Network Training Fog Computing Multilevel Model Remote Work Fog Robotics

July 26, 2024

NeuSemSlice: Towards Effective DNN Model Maintenance via Neuron-level Semantic Slicing
Shide Zhou, Tianlin Li, Yihao Huang, Ling Shi, Kailong Wang, Yang Liu, Haoyu Wang
Semantic Model Deep Neural Network Training Semantic Cue Task Specific Neuron

April 8, 2024

Slax: A Composable JAX Library for Rapid and Flexible Prototyping of Spiking Neural Networks
Thomas M. Summe, Siddharth Joshi
Neural Network Training Data Spiking Neural Network Back Propagation Deep Neural Network Training Fast Prototyping JAX Primitive

March 14, 2024

DiTMoS: Delving into Diverse Tiny-Model Selection on Microcontrollers
Xiao Ma, Shengfeng He, Hezhe Qiao, Dong Ma
Neural Network Deep Neural Network Generative Adversarial 32 Bit Microcontrollers Deep Neural Network Training Network Selection

February 14, 2024

Switch EMA: A Free Lunch for Better Flatness and Sharpness
Siyuan Li, Zicheng Liu, Juanxi Tian, Ge Wang, Zedong Wang, Weiyang Jin, Di Wu, Cheng Tan, Tao Lin, Yang Liu, Baigui Sun, Stan Z. Li
Deep Neural Network Free Lunch Deep Neural Network Training Node Feature Sharpness Document Flattening Optimal Generalization Exponential Moving Average

November 28, 2023

Spiking Neural Networks with Dynamic Time Steps for Vision Transformers
Gourav Datta, Zeyu Liu, Anni Li, Peter A. Beerel
Convolutional Neural Network Vision Transformer Spiking Neural Network Deep Neural Network Training Time Step

November 26, 2023

Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search
Zhiqi Lin, Youshan Miao, Guanbin Xu, Cheng Li, Olli Saarikivi, Saeed Maleki, Fan Yang
Deep Neural Network Deep Neural Network Training Operator Level Optimization

October 2, 2023

MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training
Daegun Yoon, Sangyoon Oh
Multiplicative Size Scaling Random Sparsification Deep Neural Network Training Gradient Sparsification Aggregated Gradient Micro Level

May 29, 2023

Trainable and Explainable Simplicial Map Neural Networks
Eduardo Paluzo-Hidalgo, Miguel A. Gutiérrez-Naranjo, Rocio Gonzalez-Diaz
Neural Network Adversarial Example High Dimensional Simplicial Complex Deep Neural Network Training

May 22, 2023

ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction
Vahid Janfaza, Shantanu Mandal, Farabi Mahmud, Abdullah Muzahid
Deep Neural Network DNN Accelerator Deep Neural Network Training Gradient Prediction

March 18, 2023

Learn, Unlearn and Relearn: An Online Learning Paradigm for Deep Neural Networks
Vijaya Raghavan T. Ramkumar, Elahe Arani, Bahram Zonooz
Deep Neural Network Strong Generalization Online Learning Unlearning Framework Harmful Unlearning Deep Neural Network Training

March 8, 2023

Model-Agnostic Federated Learning
Gianluca Mittone, Walter Riviera, Iacopo Colonnelli, Robert Birke, Marco Aldinucci
Deep Neural Network Federated Learning Deep Neural Network Training Agnostic Federated Learning

February 2, 2023

Energy Efficient Training of SNN using Local Zeroth Order Method
Bhaskar Mukhoty, Velibor Bojkovic, William de Vazelhes, Giulia De Masi, Huan Xiong, Bin Gu
Zeroth Order Neuromorphic Datasets Deep Neural Network Training Energy Efficient Training

February 1, 2023

Weight Prediction Boosts the Convergence of AdamW
Lei Guan
Deep Neural Network Early Stage Convergence Deep Neural Network Training Deep Neural Network Parameter Weight Prediction

November 14, 2022

Edge-MultiAI: Multi-Tenancy of Latency-Sensitive Deep Learning Applications on Edge
SM Zobaed, Ali Mokhtari, Jaya Prakash Champati, Mathieu Kourouma, Mohsen Amini Salehi
Extreme Edge Low Latency Edge Server Deep Neural Network Training Edge Caching Multi Tenant

October 13, 2022

Feature Reconstruction Attacks and Countermeasures of DNN training in Vertical Federated Learning
Peng Ye, Zhifeng Jiang, Wei Wang, Bo Li, Baochun Li
Vertical Federated Learning Reconstruction Attack Rapid Countermeasure Deep Neural Network Training Secure Collaborative Learning

July 22, 2022

Efficient All-reduce for Distributed DNN Training in Optical Interconnect System
Fei Dai, Yawen Chen, Zhiyi Huang, Haibo Zhang, Fangfang Zhang
Deep Neural Network Training Multi Hop All Reduce

June 3, 2022

A Survey on Computationally Efficient Neural Architecture Search
Shiqing Liu, Haoyu Zhang, Yaochu Jin
Deep Neural Network Timely Survey Neural Architecture Search Deep Neural Network Training

May 28, 2022

Fault-Aware Design and Training to Enhance DNNs Reliability with Zero-Overhead
Niccolò Cavagnero, Fernando Dos Santos, Marco Ciccone, Giuseppe Averta, Tatiana Tommasi, Paolo Rech
Deep Neural Network Training Data Deep Neural Network Training Transient Fault

April 11, 2022

SATA: Sparsity-Aware Training Accelerator for Spiking Neural Networks
Ruokai Yin, Abhishek Moitra, Abhiroop Bhattacharjee, Youngeun Kim, Priyadarshini Panda
Neural Network Spiking Neural Network Deep Neural Network Training Target Annotation Backpropagation Through Time