Multi GPU

Multi-GPU computing focuses on harnessing the combined power of multiple graphics processing units (GPUs) to accelerate computationally intensive tasks, primarily in machine learning and high-performance computing. Current research emphasizes optimizing communication between GPUs, both within a single node and across a cluster, and developing efficient algorithms and model architectures (like Graph Neural Networks and Transformers) to fully utilize this parallel processing power. These advancements are crucial for tackling increasingly complex problems in various fields, from large language model inference to high-resolution image processing and scientific simulations, by significantly reducing computation time and resource requirements.

Papers

October 9, 2024

TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, Stratos Idreos
Large Language Model Potential Scalability Pytorch Model Single GPU Production Incident LLM Training Multi GPU Training Optimization

September 6, 2024

Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study
Jianwei Zhu, Hang Yin, Peng Deng, Aline Almeida, Shunfan Zhou
Single GPU Benchmark Study Inference Task Multi GPU Test Environment

August 26, 2024

Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects
Daniele De Sensi, Lorenzo Pichetti, Flavio Vella, Tiziano De Matteis, Zebin Ren, Luigi Fusco, Matteo Turisini, Daniele Cesarini, Kurt Lust, Animesh Trivedi, Duncan Roweth, Filippo Spiga, Salvatore Di Girolamo, Torsten Hoefler
DCU Insight AQ Multi GPU Exascale Computing Cray XC40 Supercomputer

July 31, 2024

DDU-Net: A Domain Decomposition-based CNN for High-Resolution Image Segmentation on Multiple GPUs
Corné Verburg, Alexander Heinlein, Eric C. Cyr
Encoder Decoder CNN Network Internet Service Domain Land Cover Multi GPU High Resolution Image Segmentation Ultra High Resolution Image

June 16, 2024

Optimized Speculative Sampling for GPU Hardware Accelerators
Dominik Wagner, Seanie Lee, Ilja Baumann, Philipp Seeberger, Korbinian Riedhammer, Tobias Bocklet
Automatic Speech Recognition Hardware Accelerator Multi GPU Chip Memory Speculative Sampling

May 14, 2024

Hierarchical Resource Partitioning on Modern GPUs: A Reinforcement Learning Approach
Urvij Saroliya, Eishi Arima, Dai Liu, Martin Schulz
Reinforcement Learning Approach Multi GPU Computing Cluster Heterogeneous GPU

April 24, 2024

NeRF-XL: Scaling NeRFs with Multiple GPUs
Ruilong Li, Sanja Fidler, Angjoo Kanazawa, Francis Williams
Neural Radiance Field Single GPU NeRF SLAM Multi GPU

April 19, 2024

Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms
Zhongyi Lin, Ning Sun, Pallab Bhattacharya, Xizhou Feng, Louis Feng, John D. Owens
Multi GPU Performance Model Machine Learning Workload

February 10, 2024

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
Keisuke Kamahori, Yile Gu, Kan Zhu, Baris Kasikci
Mixture of Expert Fast Inference Task Offloading Multi GPU GPU Memory

February 5, 2024

Single-GPU GNN Systems: Traps and Pitfalls
Yidong Gong, Arnab Tarafder, Saima Afrin, Pradeep Kumar
Neural Network Graph Neural Network Common Pitfall Multi GPU Field Trap Accurate Training System Optimization

October 19, 2023

Cooperative Minibatching in Graph Neural Networks
Muhammed Fatih Balin, Dominique LaSalle, Ümit V. Çatalyürek
Graph Neural Network Multi GPU Independent Minibatching

July 25, 2023

Multi-GPU Approach for Training of Graph ML Models on large CFD Meshes
Sebastian Strönisch, Maximilian Sander, Andreas Knüpfer, Marcus Meyer
Training Data Surrogate Model Graph Machine Learning Multi GPU Flow Simulation

June 24, 2023

Computron: Serving Distributed Deep Learning Models with Model Parallel Swapping
Daniel Zou, Xinchen Jin, Xueyang Yu, Hao Zhang, James Demmel
Deep Learning Model Model Parallelism Multi GPU GPU Cluster Performant Neural Network

April 7, 2023

ParaGraph: Weighted Graph Representation for Performance Optimization of HPC Kernels
Ali TehraniJamsaz, Alok Mishra, Akash Dutta, Abid M. Malik, Barbara Chapman, Ali Jannesari
Runtime Analysis Multi GPU Performance Optimization High Performance Computing System Paragraph Speech OpenMP Configuration Weighted Graph

March 21, 2023

DIPPM: a Deep Learning Inference Performance Predictive Model using Graph Neural Networks
Karthick Panner Selvam, Mats Brorsson
Deep Learning Graph Neural Network Inference Latency Multi GPU Inference Performance

March 15, 2023

Machine Learning-Driven Adaptive OpenMP For Portable Performance on Heterogeneous Systems
Giorgis Georgakoudis, Konstantinos Parasyris, Chunhua Liao, David Beckingsale, Todd Gamblin, Bronis de Supinski
New Machine Multi GPU High Performance Computing System Heterogeneous System Heterogeneous Architecture OpenMP Configuration

January 30, 2023

SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms
Yaodan Xu, Jingzhou Sun, Sheng Zhou, Zhisheng Niu
Efficient Inference Large Batch Multi GPU Batch Size Automatic Batching Public Benchmark

January 1, 2023

MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs
Huaizheng Zhang, Yuanming Li, Wencong Xiao, Yizheng Huang, Xing Di, Jianxiong Yin, Simon See, Yong Luo, Chiew Tong Lau, Yang You
Comprehensive Benchmark Multi GPU Inference Workload Heterogeneous GPU GPU Architecture

November 25, 2022

Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism
Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, Bin Cui
Neural Network Transformer Model Multi GPU Expert Parallelism 3D Parallelism

November 13, 2022

Towards a Dynamic Composability Approach for using Heterogeneous Systems in Remote Sensing
Ilkay Altintas, Ismael Perez, Dmitry Mishin, Adrien Trouillaud, Christopher Irving, John Graham, Mahidhar Tatineni, Thomas DeFanti, Shawn Strande, Larry Smarr, Michael L. Norman
Remote Sensing Multi GPU Scientific Computing Composable Framework Heterogeneous System Geo Distributed

Multi GPU

Papers

TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training

Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study

Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects

DDU-Net: A Domain Decomposition-based CNN for High-Resolution Image Segmentation on Multiple GPUs

Optimized Speculative Sampling for GPU Hardware Accelerators

Hierarchical Resource Partitioning on Modern GPUs: A Reinforcement Learning Approach

NeRF-XL: Scaling NeRFs with Multiple GPUs

Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

Single-GPU GNN Systems: Traps and Pitfalls

Cooperative Minibatching in Graph Neural Networks

Multi-GPU Approach for Training of Graph ML Models on large CFD Meshes

Computron: Serving Distributed Deep Learning Models with Model Parallel Swapping

ParaGraph: Weighted Graph Representation for Performance Optimization of HPC Kernels

DIPPM: a Deep Learning Inference Performance Predictive Model using Graph Neural Networks

Machine Learning-Driven Adaptive OpenMP For Portable Performance on Heterogeneous Systems

SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms

MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs

Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism

Towards a Dynamic Composability Approach for using Heterogeneous Systems in Remote Sensing