Inference Pipeline

Inference pipelines optimize the execution of machine learning models, aiming to maximize speed, accuracy, and efficiency across diverse hardware and network environments. Current research focuses on techniques like pipeline parallelism, adaptive model selection (e.g., cascading ensembles), and quantization to reduce latency and memory consumption, particularly for large transformer-based models deployed on edge devices. These advancements are crucial for deploying computationally intensive AI applications in resource-constrained settings and improving the cost-effectiveness of large-scale machine learning systems. The resulting improvements in speed and efficiency have significant implications for real-time applications and broader accessibility of AI technologies.

Papers

October 28, 2024

An Actor-Critic Approach to Boosting Text-to-SQL Large Language Model
Ziyang Zheng, Haipeng Jing, Canyu Rui, Askar Hamdulla, Dong Wang
Large Language Model Natural Language Text to SQL Actor Critic Inference Pipeline

October 8, 2024

Fortify Your Foundations: Practical Privacy and Security for Foundation Model Deployments In The Cloud
Marcin Chrapek, Anjo Vahldiek-Oberwagner, Marcin Spoczynski, Scott Constable, Mona Vij, Torsten Hoefler
Foundation Model Security Related Cloud Computing Flawed Foundation Multimodal Foundation Model Threat Model Secure Approach Inference Pipeline

September 6, 2024

Hermes: Memory-Efficient Pipeline Inference for Large Models on Edge Devices
Xueyuan Han, Zinuo Cai, Yichu Zhang, Chongxin Fan, Junhan Liu, Ruhui Ma, Rajkumar Buyya
Transformer Based Model Large Model Edge Device Model Parallelism Pipeline Parallelism Scalable Inference Inference Pipeline

July 2, 2024

Revisiting Cascaded Ensembles for Efficient Inference
Steven Kolawole, Don Dennis, Ameet Talwalkar, Virginia Smith
Efficient Inference Adaptive Inference Inference Pipeline Parallel Inference Specific Model Cascade Ensemble

May 18, 2024

Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelines
Chaokun Chang, Eric Lo, Chunxiao Ye
Machine Learning Model Interpretable Machine Learning Random Feature Inference Pipeline Model Resilience

November 10, 2023

Automated Heterogeneous Low-Bit Quantization of Multi-Model Deep Learning Inference Pipeline
Jayeeta Mondal, Swarnava Dey, Arijit Mukherjee
Bit Quantization Multiple Deep Neural Network Single Deep Inference Pipeline Heterogeneous Quantization

November 7, 2023

Practical Performance Guarantees for Pipelined DNN Inference
Aaron Archer, Matthew Fahrbach, Kuikui Liu, Prakash Prabhu
Deep Neural Network Mixed Integer Performance Guarantee Pipeline Parallelism Optimality Gap Inference Pipeline

September 27, 2023

Exploiting the Signal-Leak Bias in Diffusion Models
Martin Nicolas Everaert, Athanasios Fitsios, Marco Bocchio, Sami Arpa, Sabine Süsstrunk, Radhakrishna Achanta
Diffusion Model Style Representation Side Channel Inference Pipeline

August 24, 2023

IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiency
Saeid Ghafouri, Kamran Razavi, Mehran Salmani, Alireza Sanaee, Tania Lorido-Botran, Lin Wang, Joseph Doyle, Pooyan Jamshidi
Inference Cost Cost Effective High Accuracy International Phonetic Alphabet Multi Model Inference Pipeline

May 16, 2023

Selective Guidance: Are All the Denoising Steps of Guided Diffusion Important?
Pareesa Ameneh Golnari, Zhewei Yao, Yuxiong He
Stable Diffusion Guided Diffusion Noise Estimation Inference Pipeline Denoising Step Noisy Computation Contrastive Expert Guidance

May 5, 2023

Otter: A Multi-Modal Model with In-Context Instruction Tuning
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, Ziwei Liu
Large Language Model Zero Shot Instruction Tuning Multi Modal Model Inference Pipeline

January 27, 2023

Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases
Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, Yuxiong He
Transformer Model Encoder Decoder Model Composable Framework Inference Pipeline Failure Explanation INT4 Quantization

June 14, 2022

Surgical Phase Recognition in Laparoscopic Cholecystectomy
Yunfan Li, Vinayak Shenoy, Prateek Prasanna, I. V. Ramakrishnan, Haibin Ling, Himanshu Gupta
Visual Recognition Action Segmentation Surgical Video Surgical Phase Recognition Inference Pipeline Surgical Workflow Analysis Laparoscopic Cholecystectomy

June 8, 2022

Memory-Oriented Design-Space Exploration of Edge-AI Hardware for XR Applications
Vivek Parmar, Syed Shakib Sarwar, Ziyun Li, Hsien-Hsin S. Lee, Barbara De Salvo, Manan Suri
Edge AI Design Space Exploration Inference Accelerator Non Volatile Memory Query Workload Inference Pipeline XR Application

February 21, 2022

Hybrid Learning for Orchestrating Deep Learning Inference in Multi-user Edge-cloud Networks
Sina Shahhosseini, Tianyi Hu, Dongjoo Seo, Anil Kanduri, Bryan Donyanavard, Amir M. Rahmani, Nikil Dutt
Computation Offloading Hybrid Learning Inference Pipeline Kernel Orchestration Computation Communication