Scalable Training

Scalable training focuses on efficiently training large and complex machine learning models, addressing computational limitations that hinder the deployment of advanced algorithms. Current research emphasizes optimizing training processes for various model architectures, including graph neural networks, large language models, and deep learning models for computer vision and recommendation systems, often employing techniques like model parallelism, efficient data handling, and adaptive learning rate scheduling. These advancements are crucial for enabling the development and application of powerful AI systems in diverse fields, from materials science and drug discovery to robotics and personalized recommendations, by making training feasible on massive datasets and utilizing large-scale computing resources.

Papers

November 12, 2024

FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training
Philip Zmushko, Aleksandr Beznosikov, Martin Takáč, Samuel Horváth
Large Language Model Low Rank Memory Efficient Gradient Projection Scalable Training Dependent Cost

November 6, 2024

Adaptive Consensus Gradients Aggregation for Scaled Distributed Training
Yoni Choukroun, Shlomi Azoulay, Pavel Kisilev
Deep Subspace Subspace Learning Aggregated Gradient Scalable Training

November 4, 2024

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
Xingtai Lv, Ning Ding, Kaiyan Zhang, Ermo Hua, Ganqu Cui, Bowen Zhou
Large Language Model Low Rank Attention Layer Scalable Training

October 27, 2024

Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, Xipeng Qiu
Language Model Feature Wise Sparse Representation Sparse Autoencoders LLaMA 3 Scalable Training

October 26, 2024

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading
Avinash Maurya, Jie Ye, M. Mustafa Rafique, Franck Cappello, Bogdan Nicolae
Large Language Model Transformer Model Task Offloading GPU Memory Scalable Training Hybrid CPU GPU

October 24, 2024

Little Giants: Synthesizing High-Quality Embedding Data at Scale
Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou
Synthetic Data Raw Data Visual Analogue Scale Synthetic Data Generation High Quality Scalable Training Mini Giant Model Produced Embeddings

June 25, 2024

Towards Efficient and Scalable Training of Differentially Private Deep Learning
Sebastian Rodriguez Beltran, Marlon Tobaben, Joonas Jälkö, Niki Loppi, Antti Honkela
High Efficiency Differential Privacy Gradient Descent Private Stochastic Differential Privacy SGD Computational Cost Scalable Training

June 17, 2024

RetinaGS: Scalable Training for Dense Scene Rendering with Billion-Scale 3D Gaussians
Bingling Li, Shengyi Chen, Luchao Wang, Kaimin Liao, Sijie Yan, Yuanjun Xiong
Gaussian Splatting 3D Gaussian Complex Scene Reconstruction Quality Gaussian Primitive Scalable Training

June 12, 2024

Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNN
Massimiliano Lupo Pasini, Jong Youl Choi, Kshitij Mehta, Pei Zhang, David Rogers, Jonghyun Bae, Khaled Z. Ibrahim, Ashwin M. Aji, Karl W. Schulz, Jorda Polo, Prasanna Balaprakash
Case Study Large Scale Convolutional Neural Network Architecture Atomistic Simulation Strong Scaling Graph Foundation Model Scalable Training

April 12, 2024

LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models
Juntaek Lim, Youngeun Kwon, Ranggi Hwang, Kiwan Maeng, G. Edward Suh, Minsoo Rhu
Differential Privacy Differential Privacy SGD Lazy Search Differential Analysis Scalable Training RecSys Challenge

April 9, 2024

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun
Language Model Full Potential Scaling Law Model Scaling Scalable Training

December 8, 2023

StructComp: Substituting Propagation with Structural Compression in Training Graph Contrastive Learning
Shengzhong Zhang, Wenjie Yang, Xinyuan Cao, Hongwei Zhang, Zengfeng Huang
Contrastive Learning Contrastive Loss Graph Contrastive Learning Low Rank Approximation Fast Propagation Scalable Training Structured Compression

October 30, 2023

A Scalable Training Strategy for Blind Multi-Distribution Noise Removal
Kevin Zhang, Sakshum Kulshrestha, Christopher Metzler
Neural Network Scalable Training

October 16, 2023

BLoad: Enhancing Neural Network Training with Efficient Sequential Data Handling
Raphael Ruschel, A. S. M. Iftekhar, B. S. Manjunath, Suya You
Neural Network Neural Network Model Scalable Training Sequential Computation Modern Deep Learning Model Training Scheme

October 13, 2023

Scalarization for Multi-Task and Multi-Domain Learning at Scale
Amelie Royer, Tijmen Blankevoort, Babak Ehteshami Bejnordi
Multi Task Learning Multi Task Visual Analogue Scale Multi Domain Multi Domain Learning Domain Network Weighted Sum Scalarization Scalable Training

September 1, 2023

SortedNet: A Scalable and Generalized Framework for Training Modular Deep Neural Networks
Mojtaba Valipour, Mehdi Rezagholizadeh, Hossein Rajabzadeh, Parsa Kavehzadeh, Marzieh Tahaei, Boxing Chen, Ali Ghodsi
Deep Neural Network Task Specific Model Auxiliary Sorting Network Rearranges Generalized Framework Scalable Training Dynamic Training Modular Deep

June 8, 2023

A Gradient-based Approach for Online Robust Deep Neural Network Training with Noisy Labels
Yifan Yang, Alec Koppel, Zheng Zhang
Noisy Label Gradient Based Gradient Method Robust Deep Scalable Training

July 22, 2022

Scalable training of graph convolutional neural networks for fast and accurate predictions of HOMO-LUMO gap in molecules
Jong Youl Choi, Pei Zhang, Kshitij Mehta, Andrew Blanchard, Massimiliano Lupo Pasini
Convolutional Neural Network Graph Drawing Gene Level GNN Molecular Graph Known Molecule Graph Convolutional Neural Network Large Scale Graph Datasets Scalable Training Large Scale GNN

June 20, 2022

From Multi-agent to Multi-robot: A Scalable Training and Evaluation Platform for Multi-robot Reinforcement Learning
Zhiuxan Liang, Jiannong Cao, Shan Jiang, Divya Saxena, Jinlin Chen, Huafeng Xu
Multi Agent Reinforcement Learning Multi Agent Multi Robot Multi Robot System Scalable Training Real World Multi Robot Evaluation Platform Multi Robot Reinforcement Learning Cooperative Lane

April 13, 2022

Scalable Training of Language Models using JAX pjit and TPUv4
Joanna Yoo, Kuba Perlin, Siddhartha Rao Kamalakara, João G. M. Araújo
Language Model Technical Challenge Training Strategy Scalable Training Edge TPU JAX Primitive

Scalable Training

Papers

FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training

Adaptive Consensus Gradients Aggregation for Scaled Distributed Training

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention

Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading

Little Giants: Synthesizing High-Quality Embedding Data at Scale

Towards Efficient and Scalable Training of Differentially Private Deep Learning

RetinaGS: Scalable Training for Dense Scene Rendering with Billion-Scale 3D Gaussians

Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNN

LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

StructComp: Substituting Propagation with Structural Compression in Training Graph Contrastive Learning

A Scalable Training Strategy for Blind Multi-Distribution Noise Removal

BLoad: Enhancing Neural Network Training with Efficient Sequential Data Handling

Scalarization for Multi-Task and Multi-Domain Learning at Scale

SortedNet: A Scalable and Generalized Framework for Training Modular Deep Neural Networks

A Gradient-based Approach for Online Robust Deep Neural Network Training with Noisy Labels

Scalable training of graph convolutional neural networks for fast and accurate predictions of HOMO-LUMO gap in molecules

From Multi-agent to Multi-robot: A Scalable Training and Evaluation Platform for Multi-robot Reinforcement Learning

Scalable Training of Language Models using JAX pjit and TPUv4