DNN Training

DNN training aims to optimize deep neural network parameters to achieve high accuracy on a given task. Current research focuses on improving training efficiency through techniques like quantization, pruning, and novel parallelization strategies (e.g., pipeline parallelism, asynchronous communication), as well as addressing challenges in resource-constrained environments (e.g., edge devices, energy harvesting systems). These advancements are crucial for deploying DNNs in various applications, from mobile devices to large-scale data centers, while mitigating the environmental impact of their substantial computational demands and enhancing model robustness against vulnerabilities like backdoor attacks.

Papers

March 4, 2023

Chasing Low-Carbon Electricity for Practical and Sustainable DNN Training
Zhenning Yang, Luoxi Meng, Jae-Won Chung, Mosharaf Chowdhury
Deep Learning Deep Neural Network Supervised ImageNet DNN Training Sustainable Deep Learning Carbon Intensity Carbon Neutrality

March 2, 2023

Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communication
Meng Zhang, Qinghao Hu, Peng Sun, Yonggang Wen, Tianwei Zhang
Token Boosting GNN Training DNN Training GNN Training Framework Full Graph

February 13, 2023

Expediting Distributed DNN Training with Device Topology-Aware Graph Deployment
Shiwei Zhang, Xiaodong Yi, Lansong Diao, Chuan Wu, Siyu Wang, Wei Lin
DNN Training Heterogeneous Cluster

February 8, 2023

Masking Kernel for Learning Energy-Efficient Representations for Speaker Recognition and Mobile Health
Apiwat Ditthapron, Emmanuel O. Agu, Adam C. Lammert
Speaker Recognition Energy Efficient Mobile Health DNN Training Efficient Representation

November 22, 2022

Self-Ensemble Protection: Training Checkpoints Are Good Data Protectors
Sizhe Chen, Geng Yuan, Xinwen Cheng, Yifan Gong, Minghai Qin, Yanzhi Wang, Xiaolin Huang
DNN Training Model Checkpoint Model Gradient Intermediate Checkpoint Ensemble Defense

November 19, 2022

Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Training
Simla Burcu Harma, Ayan Chakraborty, Nicholas Sperry, Babak Falsafi, Martin Jaggi, Yunho Oh
Low Precision Accuracy Improvement DNN Training Bit Training

October 24, 2022

Selecting and Composing Learning Rate Policies for Deep Neural Networks
Yanzhao Wu, Ling Liu
Deep Neural Network DNN Model View Selection DNN Training Learning Rate Schedule DNN Optimization

September 9, 2022

ApproxTrain: Fast Simulation of Approximate Multipliers for DNN Training and Inference
Jing Gong, Hassaan Saadat, Hasindu Gamaarachchi, Haris Javaid, Xiaobo Sharon Hu, Sri Parameswaran
Scientific Inference DNN Architecture DNN Training Fast Simulation Inference Accelerator Approximate Multiplier Neural Network Library

August 18, 2022

L3: Accelerator-Friendly Lossless Image Format for High-Resolution, High-Throughput DNN Training
Jonghyun Bae, Woohyeon Baek, Tae Jun Ham, Jae W. Lee
High Resolution DNN Training Lossy Image Compression Training Pipeline Training Throughput Neural Network Training

August 12, 2022

Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training
Jie You, Jae-Won Chung, Mosharaf Chowdhury
Deep Neural Network Human Understanding Power Consumption DNN Training Performance Optimization GPU Energy

August 5, 2022

DIVISION: Memory Efficient Training via Dual Activation Precision
Guanchu Wang, Zirui Liu, Zhimeng Jiang, Ninghao Liu, Na Zou, Xia Hu
Deep Neural Network Backward Propagation DNN Training High Compression Memory Efficient Training

June 24, 2022

QReg: On Regularization Effects of Quantization
MohammadHossein AskariHemmat, Reyhane Askari Hemmat, Alex Hoffman, Ivan Lazarevich, Ehsan Saboori, Olivier Mastropietro, Sudhakar Sah, Yvon Savaria, Jean-Pierre David
Quantization Operator Multiplier Free Quantization Weight Quantization DNN Training Quantization Level Regularization Effect

June 9, 2022

Swan: A Neural Engine for Efficient DNN Training on Smartphone SoCs
Sanjay Sri Vallabh Singapuram, Fan Lai, Chuheng Hu, Mosharaf Chowdhury
Neural Network DNN Model DNN Training Neural Engine

May 29, 2022

A General Multiple Data Augmentation Based Framework for Training Deep Neural Networks
Binyan Hu, Yu Sun, A. K. Qin
Data Augmentation Training Deep DNN Training Multiple Deep Neural Network

May 24, 2022

One-Pixel Shortcut: on the Learning Preference of Deep Neural Networks
Shutong Wu, Sizhe Chen, Cihang Xie, Xiaolin Huang
Deep Neural Network Adversarial Training Pixel Wise DNN Training Unlearnable Example Unlearnable Datasets Learning Preference

May 5, 2022

dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training
Hanpeng Hu, Chenyu Jiang, Yuchen Zhong, Yanghua Peng, Chuan Wu, Yibo Zhu, Haibin Lin, Chuanxiong Guo
Deep Learning Framework DNN Model DNN Training Performance Bottleneck Dataset Specific Profiling

April 1, 2022

Testing Feedforward Neural Networks Training Programs
Houssem Ben Braiek, Foutse Khomh
Deep Neural Network DNN Framework DNN Model DNN Training Deep Learning Testing

March 24, 2022

Repairing Group-Level Errors for DNNs Using Weighted Regularization
Ziyuan Zhong, Yuchi Tian, Conor J. Sweeney, Vicente Ordonez, Baishakhi Ray
Deep Neural Network DNN Model DNN Training Weight Regularization Error Prone Group

January 17, 2022

Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezing
Yiding Wang, Decang Sun, Kai Chen, Fan Lai, Mosharaf Chowdhury
Neural Network Deep Neural Network DNN Model Plasticity Rule DNN Training Layer Freezing

January 13, 2022

Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering
Mohammed Oualid Attaoui, Hazem Fahmy, Fabrizio Pastore, Lionel Briand
Deep Neural Network Feature Extraction DNN Model DNN Training DNN Accuracy Safety Analysis DNN Repair Retraining Method