Distributed Inference

Distributed inference aims to overcome the limitations of running large deep neural networks (DNNs) on single devices by distributing the computational load across multiple nodes. Current research focuses on optimizing model partitioning strategies (e.g., layer-wise sharding), developing communication-efficient algorithms to minimize data transfer overhead, and designing robust systems that tolerate device failures and network instability, often employing techniques like early-exit and adaptive model architectures (e.g., ResNets). This approach is crucial for enabling the deployment of powerful AI models on resource-constrained edge devices and decentralized networks, impacting fields like IoT, personalized recommendations, and large language model accessibility.

Papers

October 15, 2024

ED-ViT: Splitting Vision Transformer for Distributed Inference on Edge Devices
Xiang Liu, Yijun Song, Xia Li, Yifei Sun, Huiying Lan, Zemin Liu, Linshan Jiang, Jialin Li
Vision Transformer Deep Learning Model Transformer Model Extreme Edge Smart Device ViT Lens SNN Model Distributed Inference

August 8, 2024

Early-Exit meets Model-Distributed Inference at Edge Networks
Marco Colocrese, Erdem Koyuncu, Hulya Seferoglu
Deep Neural Network Edge Network Early eXit Inference Strategy Distributed Inference

July 29, 2024

Model Agnostic Hybrid Sharding For Heterogeneous Distributed Inference
Claudio Angione, Yue Zhao, Harry Yang, Ahmad Farhan, Fielding Johnston, James Buban, Patrick Colangelo
Artificial Intelligence Technology Large AI Model Distributed Inference Table Sharding Sequential Deep Decentralized Inference

July 25, 2024

Privacy-Preserving Hierarchical Model-Distributed Inference
Fatemeh Jafarian Dehkordi, Yasaman Keshtkarjahromi, Hulya Seferoglu
Extreme Edge Homomorphic Encryption Model Inference Distributed Inference Oblivious Transfer

January 17, 2024

Fluid Dynamic DNNs for Reliable and Adaptive Distributed Inference on Edge Devices
Lei Xun, Mingyu Hu, Hengrui Zhao, Amit Kumar Singh, Jonathon Hare, Geoff V. Merrett
Edge Device DNN Model Fluid Dynamic Distributed Inference Static Deep Neural Network

December 13, 2023

Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, Colin Raffel
Large Language Model Fine Tuning Scientific Inference NLP Task Inference Cost Distributed Inference

December 7, 2023

Moirai: Towards Optimal Placement for Distributed Inference on Heterogeneous Devices
Beibei Zhang, Hongwei Zhu, Feng Gao, Zhihui Yang, Sean Xiaoyang Wang
Deep Neural Network DNN Model Heterogeneous Device Distributed Inference Optimal Placement Device Placement

July 21, 2023

Adaptive ResNet Architecture for Distributed Inference in Resource-Constrained IoT Systems
Fazeela Mazhar Khan, Emna Baccour, Aiman Erbad, Mounir Hamdi
Deep Neural Network Resource Constrained ResNet Based ResNet Architecture Distributed Inference Adaptive Architecture

February 23, 2023

Hera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations
Yujeong Choi, John Kim, Minsoo Rhu
Personalized Recommendation Distributed Inference Baseline Recommendation Heterogeneous Memory

February 22, 2023

DISCO: Distributed Inference with Sparse Communications
Minghai Qin, Chao Sun, Jaco Hofmann, Dejan Vucinic
Deep Neural Network Scientific Inference DNN Model Distributed Inference Sparse Communication Layer Model Parallelism

December 21, 2022

Deep Reinforcement Learning for Trajectory Path Planning and Distributed Inference in Resource-Constrained UAV Swarms
Marwan Dhuheir, Emna Baccour, Aiman Erbad, Sinan Sabeeh Al-Obaidi, Mounir Hamdi
Deep Reinforcement Learning Unmanned Aerial Vehicle Trajectory Planning UAV Image UAV Swarm Distributed Inference

April 28, 2022

On the Arithmetic and Geometric Fusion of Beliefs for Distributed Inference
Mert Kayaalp, Yunus Inan, Emre Telatar, Ali H. Sayed
Learning Rate Belief State Hypothesis Testing Arithmetic Operation Distributed Inference Geometry Aware Fusion Human Belief Network

December 17, 2021

Communication-oriented Model Fine-tuning for Packet-loss Resilient Distributed Inference under Highly Lossy IoT Networks
Sohei Itahara, Takayuki Nishio, Yusuke Koda, Koji Yamamoto
Fine Tuning End to End Scientific Inference Low Latency Internet of Thing Network Distributed Inference

November 3, 2021

Communication-Efficient Separable Neural Network for Distributed Inference on Edge Devices
Jun-Liang Lin, Sheng-De Wang
Neural Network Neural Architecture Search Edge Device Model Parallelism Distributed Inference Separable Gaussian Neural Network

Distributed Inference

Papers

ED-ViT: Splitting Vision Transformer for Distributed Inference on Edge Devices

Early-Exit meets Model-Distributed Inference at Edge Networks

Model Agnostic Hybrid Sharding For Heterogeneous Distributed Inference

Privacy-Preserving Hierarchical Model-Distributed Inference

Fluid Dynamic DNNs for Reliable and Adaptive Distributed Inference on Edge Devices

Distributed Inference and Fine-tuning of Large Language Models Over The Internet

Moirai: Towards Optimal Placement for Distributed Inference on Heterogeneous Devices

Adaptive ResNet Architecture for Distributed Inference in Resource-Constrained IoT Systems

Hera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations

DISCO: Distributed Inference with Sparse Communications

Deep Reinforcement Learning for Trajectory Path Planning and Distributed Inference in Resource-Constrained UAV Swarms

On the Arithmetic and Geometric Fusion of Beliefs for Distributed Inference

Communication-oriented Model Fine-tuning for Packet-loss Resilient Distributed Inference under Highly Lossy IoT Networks

Communication-Efficient Separable Neural Network for Distributed Inference on Edge Devices