Training Optimization

Training optimization in deep learning aims to accelerate and improve the efficiency of model training, addressing the increasingly high computational costs associated with larger models and datasets. Current research focuses on optimizing backpropagation, exploring techniques like low-rank approximations and quantization to reduce computational burden, as well as developing novel training schedules and algorithms such as bilevel optimization and incorporating exemplar optimization into the training process itself. These advancements are crucial for making deep learning more accessible and practical, enabling researchers and practitioners to train sophisticated models with limited resources and accelerating progress across diverse applications.

Papers

October 9, 2024

TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, Stratos Idreos
Large Language Model Potential Scalability Pytorch Model Single GPU Production Incident LLM Training Multi GPU Training Optimization

June 21, 2024

HLQ: Fast and Efficient Backpropagation via Hadamard Low-rank Quantization
Seonggon Kim, Eunhyeok Park
Low Rank Approximation Efficient Deep Hadamard Layer Training Optimization Low Rank Quantization

March 1, 2024

Machine Learning Training Optimization using the Barycentric Correction Procedure
Sofia Ramos-Pulido, Neil Hernandez-Gress, Hector G. Ceballos-Cancino
Machine Learning Machine Learning Algorithm Radial Basis Function Barycentric Coordinate Training Optimization

January 25, 2024

Incorporating Test-Time Optimization into Training with Dual Networks for Human Mesh Recovery
Yongwei Nie, Mingxian Fan, Chengjiang Long, Qing Zhang, Jian Zhu, Xuemiao Xu
Training Data Human Mesh Recovery Dual Network Training Optimization

January 13, 2024

Joint Unsupervised and Supervised Training for Automatic Speech Recognition via Bilevel Optimization
A F M Saif, Xiaodong Cui, Han Shen, Songtao Lu, Brian Kingsbury, Tianyi Chen
Automatic Speech Recognition Speech Recognition Bilevel Optimization Acoustic Model Supervised Training Training Optimization

June 11, 2023

Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute
William Chen, Xuankai Chang, Yifan Peng, Zhaoheng Ni, Soumi Maiti, Shinji Watanabe
Self Supervised Learning Self Supervised Pre Trained Speech Processing Knowledge Barrier ASR Model Training Optimization

February 16, 2023

Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Training
Hongzheng Chen, Cody Hao Yu, Shuai Zheng, Zhen Zhang, Zhiru Zhang, Yida Wang
Pytorch Model Model Training Task Scheduling Sequential Optimization Kernel Performance Training Optimization DeepSpeed Data Efficiency

December 14, 2022

APOLLO: An Optimized Training Approach for Long-form Numerical Reasoning
Jiashuo Sun, Hang Zhang, Chen Lin, Xiangdong Su, Yeyun Gong, Jian Guo
Negative Sampling Numerical Reasoning Logic Program Training Optimization APOLLO Apollonion ICARUS Retriever Generator

December 13, 2022

Improving Depression estimation from facial videos with face alignment, training optimization and scheduling
Manuel Lage Cañellas, Constantino Álvarez Casado, Le Nguyen, Miguel Bordallo López
Deep Learning Model Task Scheduling 3d CNN Facial Video Face Alignment Training Optimization Score Level Fusion

June 20, 2022

Deep Learning Models on CPUs: A Methodology for Efficient Training
Quchen Fu, Ramesh Chukka, Keith Achorn, Thomas Atta-fosu, Deepak R. Canchi, Zhongwei Teng, Jules White, Douglas C. Schmidt
Deep Learning Model NCD Method Pytorch Model Efficient Training Novel Aerial Inspection RetinaNet Training Optimization

Training Optimization

Papers

TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training

HLQ: Fast and Efficient Backpropagation via Hadamard Low-rank Quantization

Machine Learning Training Optimization using the Barycentric Correction Procedure

Incorporating Test-Time Optimization into Training with Dual Networks for Human Mesh Recovery

Joint Unsupervised and Supervised Training for Automatic Speech Recognition via Bilevel Optimization

Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute

Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Training

APOLLO: An Optimized Training Approach for Long-form Numerical Reasoning

Improving Depression estimation from facial videos with face alignment, training optimization and scheduling

Deep Learning Models on CPUs: A Methodology for Efficient Training