Training Bottleneck

Training bottlenecks in machine learning represent limitations in the training process that hinder model performance or efficiency. Current research focuses on mitigating these bottlenecks across various model types, including recurrent neural networks, transformers, and deep reinforcement learning agents, employing techniques like sharpness-aware minimization, data pipeline optimization, and novel architectural designs (e.g., incorporating recurrent mechanisms into transformers or using DFT output layers). Overcoming these bottlenecks is crucial for advancing the field, enabling the training of larger, more complex models and improving the efficiency of existing models for diverse applications ranging from recommendation systems to medical image reconstruction.

Papers

October 25, 2024

Analyzing Neural Network Robustness Using Graph Curvature
Shuhang Tan, Jayson Sia, Paul Bogdan, Radoslav Ivanov
Neural Network Robustness Representation Bottleneck Robustness Problem Ricci Flow Graph Curvature Training Bottleneck

May 5, 2024

RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation
Zelei Cheng, Xian Wu, Jiahao Yu, Sabrina Yang, Gang Wang, Xinyu Xing
Reinforcement Learning Deep Reinforcement Learning Line by Line Explanation Sparse Reward Agent Performance Training Bottleneck Rice Kernel Initial State Distribution

February 1, 2024

Investigating Recurrent Transformers with Dynamic Halt
Jishnu Ray Chowdhury, Cornelia Caragea
Recurrent Transformer Recurrence Relation Training Bottleneck Dynamic Token Halting

January 14, 2024

Inroads to a Structured Data Natural Language Bijection and the role of LLM annotation
Blake Vente
Multi Task Integral Role Sequence to Sequence Cross Task Generalization Inverse Task LLM Annotation Training Bottleneck

October 16, 2023

Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification
Andreas Grivas, Antonio Vergari, Adam Lopez
Many Sparse Multi Label Classification Sigmoid Function Label Set Training Bottleneck

September 21, 2023

Parallelizing non-linear sequential models over the sequence length
Yi Heng Lim, Qi Zhu, Joshua Selfridge, Muhammad Firmansyah Kasim
Recurrent Neural Network Gated Recurrent Unit Sequential Model Sequence Length Training Bottleneck

August 15, 2023

ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognition
Yixuan Zhou, Yi Qu, Xing Xu, Hengtao Shen
Class Imbalance Sharpness Aware Minimization Glance Annotation Long Tailed Classification Imbalanced Medical Image Training Bottleneck

August 13, 2023

InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models
Kabir Nagrecha, Lingyi Liu, Pablo Delgado, Prasanna Padmanabhan
Efficient Pipeline Training Bottleneck DL Model Deep Recommendation

November 29, 2022

Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference
Yabin Wang, Zhiheng Ma, Zhiwu Huang, Yaowei Wang, Zhou Su, Xiaopeng Hong
Incremental Learning Two Stage Catastrophic Interference Hui Walter Paradigm Social Isolation Incremental Learning Framework Training Bottleneck Fairness Aware Aggregation

October 20, 2022

Physics-informed Deep Diffusion MRI Reconstruction with Synthetic Data: Break Training Data Bottleneck in Artificial Intelligence
Chen Qian, Yuncheng Gao, Mingyang Han, Zi Wang, Dan Ruan, Yu Shen, Yaping Wu, Yirong Zhou, Chengyan Wang, Boyu Jiang, Ran Tao, Zhigang Wu, Jiazheng Wang, Liuhong Zhu, Yi Guo, Taishan Kang, Jianzhong Lin, Tao Gong, Chen Yang, Guoqiang Fei, Meijin Lin, Di Guo, Jianjun Zhou, Meiyun Wang, Xiaobo Qu
Artificial Intelligence Synthetic Data Magnetic Resonance Imaging Diffusion MRI Vivo Imaging Training Bottleneck

May 30, 2022

Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning
Aniket Didolkar, Kshitij Gupta, Anirudh Goyal, Nitesh B. Gundavarapu, Alex Lamb, Nan Rosemary Ke, Yoshua Bengio
Recurrent Neural Network Generalization Performance Critical Synthesis Level Representation Compressed Representation Sequence Learning Training Bottleneck

February 17, 2022

Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines
Alexander Isenko, Ruben Mayer, Jeffrey Jedele, Hans-Arno Jacobsen
Deep Learning Potential Scalability Worst Case User Throughput Preprocessing Method Training Bottleneck

November 18, 2021

Automatic Neural Network Pruning that Efficiently Preserves the Model Accuracy
Thibault Castells, Seul-Ki Yeom
Neural Network Pruning Method Model Accuracy Floating Point Neural Network Performance Training Bottleneck Automatic Pruning

Training Bottleneck

Papers

Analyzing Neural Network Robustness Using Graph Curvature

RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation

Investigating Recurrent Transformers with Dynamic Halt

Inroads to a Structured Data Natural Language Bijection and the role of LLM annotation

Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification

Parallelizing non-linear sequential models over the sequence length

ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognition

InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models

Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference

Physics-informed Deep Diffusion MRI Reconstruction with Synthetic Data: Break Training Data Bottleneck in Artificial Intelligence

Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning

Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines

Automatic Neural Network Pruning that Efficiently Preserves the Model Accuracy