Inference Service

Inference services, which deliver predictions from machine learning models, are a critical component of many applications, aiming to optimize accuracy, latency, and cost. Current research focuses on improving efficiency through techniques like model cascades, dynamic modality selection, and optimized resource allocation across heterogeneous hardware, as well as enhancing privacy using secure multi-party computation and novel data protection methods. These advancements are crucial for deploying large language models and other complex AI systems in resource-constrained environments and sensitive applications, impacting both the scalability of AI and its responsible use.

Papers

October 31, 2024

October 28, 2024

FusedInf: Efficient Swapping of DNN Models for On-Demand Serverless Inference Services on the Edge
Sifat Ut Taki, Arthi Padmanabhan, Spyridon Mastorakis
DNN Model Edge AI Swap Distance Minimization Serverless Computing Inference Service

October 24, 2024

BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching
Peizhuang Cong, Qizhi Chen, Haochen Zhao, Tong Yang
Full Model LLM Inference Large Language Inference Service Automatic Batching Batch Inference

August 7, 2024

MPC-Minimized Secure LLM Inference
Deevashwer Rathee, Dacheng Li, Ion Stoica, Hao Zhang, Raluca Popa
Scientific Inference LLM Inference Secure Multi Party Computation Inference Service Secure Inference

June 20, 2024

CascadeServe: Unlocking Model Cascades for Inference Serving
Ferdi Kossmann, Ziniu Wu, Alex Turk, Nesime Tatbul, Lei Cao, Samuel Madden
Efficient Inference Cascade Model TF Cascade Inference Workload Inference Service Model Replication

June 3, 2024

PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration
Ziqian Zeng, Jianwei Wang, Junyao Yang, Zhengdong Lu, Huiping Zhuang, Cen Chen
Scientific Inference LLM Inference Restoration Quality Inference Service Privacy Level Privacy Recovery

February 24, 2024

Selective Task offloading for Maximum Inference Accuracy and Energy efficient Real-Time IoT Sensing Systems
Abdelkarim Ben Sada, Amar Khelloufi, Abdenacer Naouri, Huansheng Ning, Sahraoui Dhelim
Energy Policy Research Inference Efficiency Model Inference Internet of Thing Environment Edge Inference Inference Service Probabilistic Learning

January 1, 2024

SecFormer: Towards Fast and Accurate Privacy-Preserving Inference for Large Language Models
Jinglong Luo, Yehong Zhang, Zhuo Zhang, Jiaqi Zhang, Xin Mu, Hui Wang, Yue Yu, Zenglin Xu
Large Language Model Transformer Architecture Secure Multi Party Computation Inference Service Privacy Preserving Inference

November 7, 2023

Prompt Cache: Modular Attention Reuse for Low-Latency Inference
In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, Lin Zhong
Large Language Model Inference Service GPU Inference Based Inference

November 3, 2023

ERASER: Machine Unlearning in MLaaS via an Inference Serving-Aware Approach
Yuke Hu, Jian Lou, Jiaqi Liu, Wangze Ni, Feng Lin, Zhan Qin, Kui Ren
Machine Unlearning Unlearning Framework Inference Framework Inference Service Separable Multi Concept ERASER

October 27, 2023

MOSEL: Inference Serving Using Dynamic Modality Selection
Bodun Hu, Le Xu, Jeongyoon Moon, Neeraja J. Yadwadkar, Aditya Akella
Scientific Inference Input Modality Inference Service Modality Selection Modality Compensation Multi Modal Machine Learning

May 10, 2023

Fast Distributed Inference Serving for Large Language Models
Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin
Large Language Model Scientific Inference LLM Inference Model Inference Inference Service

April 21, 2023

Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems
Mehran Salmani, Saeid Ghafouri, Alireza Sanaee, Kamran Razavi, Max Mühlhäuser, Joseph Doyle, Pooyan Jamshidi, Mohsen Sharifi
Machine Learning Model Low Latency Accuracy Improvement Cost Effective Inference Service Resource Scaling

March 31, 2023

A Generative Framework for Low-Cost Result Validation of Machine Learning-as-a-Service Inference
Abhinav Kumar, Miguel A. Guirao Aguilera, Reza Tourani, Satyajayant Misra
Generative Framework Efficient Verification Inference Service Integrity Verification Client Side

March 27, 2023

Privacy-preserving machine learning for healthcare: open challenges and future perspectives
Alejandro Guerra-Manzanares, L. Julian Lechuga Lopez, Michail Maniatakos, Farah E. Shamout
Machine Learning New Machine Healthcare System Open Challenge Medical Data Future Perspective Inference Service Privacy Preserving Training Healthcare Prediction Task

November 2, 2022

MPCFormer: fast, performant and private Transformer inference with MPC
Dacheng Li, Rulin Shao, Hongyi Wang, Han Guo, Eric P. Xing, Hao Zhang
Inference Latency Transformer Inference Private Inference Inference Service Multi Party Computation Sl MPC

October 17, 2022

A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models
Yingcan Wei, Matthias Langer, Fan Yu, Minseok Lee, Kingsley Liu, Jerry Shi, Joey Wang
Large Model Modern Recommender System Inference Service

September 14, 2022

SEEK: model extraction attack against hybrid secure inference protocols
Si Chen, Junfeng Fan
Homomorphic Encryption Model Extraction Attack Inference Service Party Computation Secure Inference

July 23, 2022

RIBBON: Cost-Effective and QoS-Aware Deep Learning Model Inference using a Diverse Pool of Cloud Computing Instances
Baolin Li, Rohan Basu Roy, Tirthak Patel, Vijay Gadepally, Karen Gettings, Devesh Tiwari
Scientific Inference Inference Cost Cost Effective Multiple Instance Model Inference Inference Service Diverse Body

Inference Service

Papers

MESS+: Energy-Optimal Inferencing in Language Model Zoos with Service Level Guarantees

Kernel Looping: Eliminating Synchronization Boundaries for Peak Inference Performance

FusedInf: Efficient Swapping of DNN Models for On-Demand Serverless Inference Services on the Edge

BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching

MPC-Minimized Secure LLM Inference

CascadeServe: Unlocking Model Cascades for Inference Serving

PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration

Selective Task offloading for Maximum Inference Accuracy and Energy efficient Real-Time IoT Sensing Systems

SecFormer: Towards Fast and Accurate Privacy-Preserving Inference for Large Language Models

Prompt Cache: Modular Attention Reuse for Low-Latency Inference

ERASER: Machine Unlearning in MLaaS via an Inference Serving-Aware Approach

MOSEL: Inference Serving Using Dynamic Modality Selection

Fast Distributed Inference Serving for Large Language Models

Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems

A Generative Framework for Low-Cost Result Validation of Machine Learning-as-a-Service Inference

Privacy-preserving machine learning for healthcare: open challenges and future perspectives

MPCFormer: fast, performant and private Transformer inference with MPC

A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models

SEEK: model extraction attack against hybrid secure inference protocols

RIBBON: Cost-Effective and QoS-Aware Deep Learning Model Inference using a Diverse Pool of Cloud Computing Instances