DNN Training

DNN training aims to optimize deep neural network parameters to achieve high accuracy on a given task. Current research focuses on improving training efficiency through techniques like quantization, pruning, and novel parallelization strategies (e.g., pipeline parallelism, asynchronous communication), as well as addressing challenges in resource-constrained environments (e.g., edge devices, energy harvesting systems). These advancements are crucial for deploying DNNs in various applications, from mobile devices to large-scale data centers, while mitigating the environmental impact of their substantial computational demands and enhancing model robustness against vulnerabilities like backdoor attacks.

Papers

November 13, 2024

DNN Task Assignment in UAV Networks: A Generative AI Enhanced Multi-Agent Reinforcement Learning Approach
Xin Tang, Qian Chen, Wenjie Weng, Binhan Liao, Jiacheng Wang, Xianbin Cao, Xiaohuan Li
Multi Agent Reinforcement Learning Generative AI Unmanned Aerial Vehicle UAV Network DNN Training UAV Swarm

October 7, 2024

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse
Arthur Jacot, Peter Súkeník, Zihan Wang, Marco Mondelli
Deep Neural Network Neural Collapse Wide Neural Network DNN Training Unconstrained Feature Model Variability Collapse

August 25, 2024

Revisiting DNN Training for Intermittently Powered Energy Harvesting Micro Computers
Cyan Subhra Mishra, Deeksha Chaudhary, Jack Sampson, Mahmut Taylan Knademir, Chita Das
DNN Training Energy Harvesting Invariant Dropout

July 15, 2024

On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers
Mark Deutel, Frank Hannig, Christopher Mutschler, Jürgen Teich
Deep Neural Network DNN Framework Device Training DNN Training 32 Bit Microcontrollers

July 2, 2024

QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices
Juntao Zhao, Borui Wan, Yanghua Peng, Haibin Lin, Yibo Zhu, Chuan Wu
DNN Training GPU Inference

June 24, 2024

GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism
Byungsoo Jeon, Mengdi Wu, Shiyi Cao, Sunghyun Kim, Sunghyun Park, Neeraj Aggarwal, Colin Unger, Daiyaan Arfeen, Peiyuan Liao, Xupeng Miao, Mohammad Alizadeh, Gregory R. Ganger, Tianqi Chen, Zhihao Jia
System Performance Potential Scalability DNN Training Pipeline Parallelism Neural Network Training

March 4, 2024

Better Schedules for Low Precision Training of Deep Neural Networks
Cameron R. Wolfe, Anastasios Kyrillidis
Deep Neural Network DNN Training Efficient Scheduling Low Precision Training Accurate Training

February 14, 2024

MLTCP: Congestion Control for DNN Training
Sudarsanan Rajasekaran, Sanjoli Narang, Anton A. Zabreyko, Manya Ghobadi
DNN Training GPU Cluster Congestion Control

November 29, 2023

Mirage: An RNS-Based Photonic Accelerator for DNN Training
Cansu Demirkiran, Guowei Yang, Darius Bunandar, Ajay Joshi
Low Precision DNN Training Photonic Accelerator Photonic Computing Photonic Tensor Core

November 22, 2023

Comprehensive Evaluation of GNN Training Systems: A Data Management Perspective
Hao Yuan, Yajiong Liu, Yanfeng Zhang, Xin Ai, Qiange Wang, Chaoyi Chen, Yu Gu, Ge Yu
Graph Neural Network Comprehensive Evaluation DNN Training

November 13, 2023

Tabdoor: Backdoor Vulnerabilities in Transformer-based Neural Networks for Tabular Data
Bart Pleiter, Behrad Tajalli, Stefanos Koffas, Gorka Abad, Jing Xu, Martha Larson, Stjepan Picek
Deep Neural Network Backdoor Attack Tabular Data DNN Training Transformer Based Deep Transformer Based Neural Network

September 16, 2023

Rethinking Learning Rate Tuning in the Era of Large Language Models
Hongpeng Jin, Wenqi Wei, Xuyu Wang, Wenbin Zhang, Yanzhao Wu
Learning Rate New Era LLM Fine Tuning DNN Training Learning Rate Schedule

September 13, 2023

DNNShifter: An Efficient DNN Pruning System for Edge Computing
Bailey J. Eccles, Philip Rodgers, Peter Kilpatrick, Ivor Spence, Blesson Varghese
Edge Computing DNN Model DNN Training Deep Neural Network Parameter Pruned Model DNN Pruning

July 16, 2023

Accelerating Distributed ML Training via Selective Synchronization
Sahil Tyagi, Martin Swany
Deep Neural Network Federated Averaging Synchronization Parameter Update Barrier DNN Training Optimal Model Asynchronous Parallel

May 30, 2023

Understanding temporally weakly supervised training: A case study for keyword spotting
Heinrich Dinkel, Weiji Zhuang, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Yujun Wang
Case Study Keyword Spotting Weakly Supervised Learning Weak Label DNN Training Keyword Enrollment Supervised Training Explicit Supervision

May 12, 2023

DAISM: Digital Approximate In-SRAM Multiplier-based Accelerator for DNN Training and Inference
Lorenzo Sonnino, Shaswot Shresthamali, Yuan He, Masaaki Kondo
Scientific Inference Hardware Accelerator DNN Training Parallel Processing Processing in Memory SRAM Cell Memory Technology Common Belief

April 26, 2023

Concept-Monitor: Understanding DNN training through individual neurons
Mohammad Ali Khan, Tuomas Oikarinen, Tsui-Wei Weng
Adversarial Training Black Box High Impact Concept DNN Training Individual Neuron Conceptual Diversity

April 25, 2023

Learning Trajectories are Generalization Indicators
Jingwen Fu, Zhizheng Zhang, Dacheng Yin, Yan Lu, Nanning Zheng
Strong Generalization Generalization Error DNN Training Generalization Measure Trajectory Generalization

April 24, 2023

Survey on Unsupervised Domain Adaptation for Semantic Segmentation for Visual Perception in Automated Driving
Manuel Schwonberg, Joshua Niemeijer, Jan-Aike Termöhlen, Jörg P. Schäfer, Nico M. Schmidt, Hanno Gottschalk, Tim Fingscheidt
Deep Neural Network Timely Survey Semantic Segmentation Domain Adaptation Unsupervised Domain Adaptation Automated Driving Visual Perception DNN Training

April 18, 2023

Understand Data Preprocessing for Effective End-to-End Training of Deep Neural Networks
Ping Gong, Yuxin Ma, Cheng Li, Xiaosong Ma, Sam H. Noh
Deep Neural Network Raw Data High Fidelity DNN Training Data Preprocessing End to End Training Training Framework State of the Art NVIDIA

DNN Training

Papers

DNN Task Assignment in UAV Networks: A Generative AI Enhanced Multi-Agent Reinforcement Learning Approach

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse

Revisiting DNN Training for Intermittently Powered Energy Harvesting Micro Computers

On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers

QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices

GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism

Better Schedules for Low Precision Training of Deep Neural Networks

MLTCP: Congestion Control for DNN Training

Mirage: An RNS-Based Photonic Accelerator for DNN Training

Comprehensive Evaluation of GNN Training Systems: A Data Management Perspective

Tabdoor: Backdoor Vulnerabilities in Transformer-based Neural Networks for Tabular Data

Rethinking Learning Rate Tuning in the Era of Large Language Models

DNNShifter: An Efficient DNN Pruning System for Edge Computing

Accelerating Distributed ML Training via Selective Synchronization

Understanding temporally weakly supervised training: A case study for keyword spotting

DAISM: Digital Approximate In-SRAM Multiplier-based Accelerator for DNN Training and Inference

Concept-Monitor: Understanding DNN training through individual neurons

Learning Trajectories are Generalization Indicators

Survey on Unsupervised Domain Adaptation for Semantic Segmentation for Visual Perception in Automated Driving

Understand Data Preprocessing for Effective End-to-End Training of Deep Neural Networks