Parallel Training

Parallel training aims to accelerate the computationally intensive process of training large machine learning models by distributing the workload across multiple processors or devices. Current research focuses on optimizing this process for various model architectures, including large language models (LLMs) and convolutional neural networks (CNNs), through techniques like model and data parallelism, along with strategies to mitigate communication bottlenecks and hardware failures. Efficient parallel training is crucial for advancing the capabilities of AI systems, enabling the development and deployment of larger, more powerful models for diverse applications while reducing training time and costs.

Papers

November 22, 2023

Confidant: Customizing Transformer-based LLMs via Collaborative Edge Training
Yuhao Chen, Yuxuan Yan, Qianqian Yang, Yuanchao Shu, Shibo He, Jiming Chen
Large Language Model Natural Language Processing Transformer Based Large Language Model Parallel Training Collaborative Training Collaborative Edge Training

November 8, 2023

Fuzzy Ensembles of Reinforcement Learning Policies for Robotic Systems with Varied Parameters
Abdel Gafoor Haddad, Mohammed B. Mohiuddin, Igor Boiko, Yahya Zweiri
Reinforcement Learning Robotic System Many Parameter Reinforcement Learning Agent Parallel Training Classical Control Ensemble Aggregation

November 1, 2023

Data Augmentation for Code Translation with Comparable Corpora and Multiple References
Yiqing Xie, Atharva Naik, Daniel Fried, Carolyn Rose
Data Augmentation Data Augmentation Technique Parallel Training Code Translation Reference Translation

October 30, 2023

ViR: Towards Efficient Vision Retention Backbones
Ali Hatamizadeh, Michael Ranzinger, Shiyi Lan, Jose M. Alvarez, Sanja Fidler, Jan Kautz
Vision Transformer Computer Vision Model Parallel Training Vision Backbone Retentive Network

October 24, 2023

Accelerating Split Federated Learning over Wireless Communication Networks
Ce Xu, Jinxuan Li, Yuan Liu, Yushi Ling, Miaowen Wen
Deep Neural Network Split Learning Parallel Training Split Federated Learning Model Splitting Model Partition

October 23, 2023

ADMM Training Algorithms for Residual Networks: Convergence, Complexity and Parallel Training
Jintao Xu, Yifei Li, Wenxun Xing
Early Stage Convergence Complexity Matter Parallel Training ADMM Algorithm Unfolded D ADMM

October 22, 2023

A Quadratic Synchronization Rule for Distributed Deep Learning
Xinran Gu, Kaifeng Lyu, Sanjeev Arora, Jingzhao Zhang, Longbo Huang
Distributed Learning Parallel Training Local Gradient Local SGD

August 4, 2023

Sinhala-English Parallel Word Dictionary Dataset
Kasun Wickramasinghe, Nisansa de Silva
Natural Language Processing Parallel Training Multilingual Task

July 26, 2023

Multi3WOZ: A Multilingual, Multi-Domain, Multi-Parallel Dataset for Training and Evaluating Culturally Adapted Task-Oriented Dialog Systems
Songbo Hu, Han Zhou, Mete Hergul, Milan Gritta, Guchun Zhang, Ignacio Iacobacci, Ivan Vulić, Anna Korhonen
Training Data Task Oriented Parallel Training Task Oriented Dialog System Task Oriented Dialog

July 5, 2023

Improving Automatic Parallel Training via Balanced Memory Workload Optimization
Yujie Wang, Youhe Jiang, Xupeng Miao, Fangcheng Fu, Shenhan Zhu, Xiaonan Nie, Yaofeng Tu, Bin Cui
Transformer Model Parallel Training GPU Memory Expert Parallelism Memory Optimization Large Scale Deep Learning

May 22, 2023

A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs
Siddharth Singh, Prajwal Singhania, Aditya K. Ranjan, Zack Sating, Abhinav Bhatele
Single GPU Parallel Training Parallel Processing Performance Bottleneck Hybrid Algorithm Data Parallelism

March 24, 2023

Scaling Expert Language Models with Unsupervised Domain Discovery
Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, Luke Zettlemoyer
Large Language Model Large Corpus Parallel Training Expert Language Model Domain Discovery

February 13, 2023

A Domain Decomposition-Based CNN-DNN Architecture for Model Parallel Training Applied to Image Recognition Problems
Axel Klawonn, Martin Lanser, Janine Weber
Convolutional Neural Network Deep Neural Network CNN Model CNN Architecture Internet Service Domain Parallel Training Convolutional Neural Network Architecture Local Convolution

February 11, 2023

Communication and Storage Efficient Federated Split Learning
Yujia Mu, Cong Shen
Timely Communication Split Learning Parallel Training Federated Split Learning

February 10, 2023

Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Training
Siddharth Singh, Abhinav Bhatele
Neural Network Model Training Parallel Training Sparse Subnetworks Sparse Matrix Exploiting Sparsity Layer Model Parallelism

January 6, 2023

Does compressing activations help model parallel training?
Song Bian, Dacheng Li, Hongyi Wang, Eric P. Xing, Shivaram Venkataraman
Parallel Training Model Parallelism Transformer Training Large Scale Transformer Model Activation Compression

December 9, 2022

Scalable Graph Convolutional Network Training on Distributed-Memory Systems
Gunduz Vehbi Demirci, Aparajita Haldar, Hakan Ferhatosmanoglu
Graph Convolutional Network Parallel Training Graph Partitioning Billion Scale Graph Shared Memory

November 28, 2022

AcceRL: Policy Acceleration Framework for Deep Reinforcement Learning
Hongjie Zhang
Reinforcement Learning Deep Reinforcement Learning Policy Learning Neural Network Compression Parallel Training

November 17, 2022

Reducing Hallucinations in Neural Machine Translation with Feature Attribution
Joël Tang, Marina Fomicheva, Lucia Specia
Loss Function Language Generation Neural Machine Translation Feature Attribution Neural Machine Translation Model Parallel Training Reducing Hallucination

November 8, 2022

Simulation-Based Parallel Training
Lucas Meyer, Alejandro Ribés, Bruno Raffin
Neural Architecture Numerical Simulation Parallel Training Complex Dynamical System Global Attractor Chaotic Lorenz