GPU Inference

GPU inference focuses on optimizing the execution of deep learning models, particularly large language models (LLMs) and convolutional neural networks (CNNs), on graphics processing units to achieve faster processing speeds and lower latency. Current research emphasizes techniques like model quantization (including novel dense-and-sparse methods), efficient memory management (e.g., leveraging CPU and NVMe memory alongside GPUs), and intelligent scheduling algorithms to maximize GPU utilization and minimize energy consumption across diverse hardware configurations. These advancements are crucial for deploying computationally intensive AI models in resource-constrained environments and for improving the scalability and cost-effectiveness of AI applications.

Papers

October 29, 2024

Pushing the Performance Envelope of DNN-based Recommendation Systems Inference on GPUs
Rishabh Jain, Vivek M. Bhasi, Adwait Jog, Anand Sivasubramaniam, Mahmut T. Kandemir, Chita R. Das
Deep Learning Scientific Inference LLM Inference Single GPU Performance Improvement GPU Inference

September 23, 2024

Benchmarking Edge AI Platforms for High-Performance ML Inference
Rakshith Jayanth, Neelesh Gupta, Viktor Prasanna
Edge Computing Edge AI Processing Unit Deep Learning Workload GPU Inference

July 29, 2024

Simply Trainable Nearest Neighbour Machine Translation with GPU Inference
Hossam Amer, Abdelrahman Abouelenin, Mohamed Maher, Evram Narouz, Mohamed Afify, Hany Awadallah
Token Level Pre Trained Transformer GPU Inference Nearest Neighbor Machine Translation

July 20, 2024

GreenStableYolo: Optimizing Inference Time and Image Quality of Text-to-Image Generation
Jingzhi Gong, Sisi Li, Giordano d'Aloisio, Zishuo Ding, Yulong Ye, William B. Langdon, Federica Sarro
Text to Image Generation Image Quality Inference Time GPU Inference

July 2, 2024

QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices
Juntao Zhao, Borui Wan, Yanghua Peng, Haibin Lin, Yibo Zhu, Chuan Wu
DNN Training GPU Inference

January 16, 2024

Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models
Shuming Shi, Enbo Zhao, Deng Cai, Leyang Cui, Xinting Huang, Huayang Li
Large Language Model Inference Engine GPU Inference

November 7, 2023

Prompt Cache: Modular Attention Reuse for Low-Latency Inference
In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, Lin Zhong
Large Language Model Inference Service GPU Inference Based Inference

October 4, 2023

From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference
Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, Vijay Gadepally
Large Language Model LLM Inference Word List Large Language Model Inference Energy Efficient Inference Inference Performance GPU Inference Energy Cost

June 13, 2023

SqueezeLLM: Dense-and-Sparse Quantization
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer
Generative Modeling Generative Large Language Model GPU Inference Squeeze Flow Quantization Performance

December 15, 2022

A Study on the Intersection of GPU Utilization and CNN Inference
Jack Kosaian, Amar Phanishayee
Convolutional Neural Network Study Feature Intersection Scenario CNN Inference GPU Implementation GPU Inference

July 20, 2022

EASNet: Searching Elastic and Accurate Network Architecture for Stereo Matching
Qiang Wang, Shaohuai Shi, Kaiyong Zhao, Xiaowen Chu
Neural Architecture Search Stereo Matching Scene Flow Dense Prediction Task GPU Inference Elastic Band

June 30, 2022

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
Transformer Model Transformer Based Model Efficient Inference Transformer Inference Neural Network Inference Global Scale GPU Inference

February 27, 2022

PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Servers
Yunseong Kim, Yujeong Choi, Minsoo Rhu
Machine Learning Augmented Reality Multi GPU GPU Inference GPU Architecture

February 16, 2022

Aryl: An Elastic Cluster Scheduler for Deep Learning
Jiamin Li, Hong Xu, Yibo Zhu, Zherui Liu, Chuanxiong Guo, Cong Wang
Deep Learning Chemical Compound GPU Inference Cluster Scheduling

GPU Inference

Papers

Pushing the Performance Envelope of DNN-based Recommendation Systems Inference on GPUs

Benchmarking Edge AI Platforms for High-Performance ML Inference

Simply Trainable Nearest Neighbour Machine Translation with GPU Inference

GreenStableYolo: Optimizing Inference Time and Image Quality of Text-to-Image Generation

QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices

Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models

Prompt Cache: Modular Attention Reuse for Low-Latency Inference

From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference

SqueezeLLM: Dense-and-Sparse Quantization

A Study on the Intersection of GPU Utilization and CNN Inference

EASNet: Searching Elastic and Accurate Network Architecture for Stereo Matching

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Servers

Aryl: An Elastic Cluster Scheduler for Deep Learning