Distributed Inference

Ask about this topic

Overview

Distributed inference studies how to divide or coordinate model execution across GPUs, edge devices, robots, sensors, and cloud services when one machine is insufficient or unsuitable. Common approaches include pipeline and tensor parallelism, mixture-of-experts placement, edge–cloud partitioning, federated or collaborative prediction, and communication-aware scheduling of requests and cached state. Systems are increasingly designed for heterogeneous hardware and changing network conditions, with simulation and real-cluster benchmarks used to assess latency, throughput, accuracy, and resource use. The central tension is that distributing computation can add communication, coordination, and reliability costs, so balancing speed and scale against privacy, consistency, and robustness remains an open challenge.

Latest papers

CardsList

75 papers8 papers in the last four weeks, down 20% on the four weeks before. 0.0% of all new papers.

  1. InferSlice: A Reference Architecture for SLO-driven LLM Inference over a Telco Wide-Area Network

    Oct 23, 2026Juan Parra-Ullauri, Anthony Chiu, Hari Madhukumar +6LLM InferenceLLM Inference Serving

    BT Group · University of Bristol

  2. Consensus-Based Distributed Filtering for Angles-Only Tracking in Cislunar Space

    Oct 22, 2026Giovanni Romagnoli, Fabrizio Pilone, Giordana BucchioniState TrackingDistributed Inference

    Department of Information Engineering, University of Pisa, Pisa 56122, Italy

  3. Minions: A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices

    Oct 9, 2026Jianhua Li, Yun Xing, Girisha De Silva +3Distributed InferenceGPU Acceleration

    Apple USA

  4. Deploying RDMA for Distributed Inference and Training At Scale

    Oct 8, 2026Yibo Huang, Yi Cui, Yukai Huang +12Communication-Efficient Distributed TrainingDistributed Inference

    Fudan University · ByteDance Inc. · University of Michigan

  5. Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure

    Sep 30, 2026Matias Parij, Pawan Paudel, Tate Berenbaum +1LLM ServingLLM Inference Scheduling

    Not Community Labs Inc. · Intel Corporation

  6. Purlin: Separating Orchestration from the Datapath of Collectives

    Sep 29, 2026Osayamen Jonathan Aimuyo, Swapnil Gandhi, Christos KozyrakisLLM Inference EfficiencyHigh-Performance Computing

    Stanford University · NVIDIA & Stanford University

  7. Reliability Testing of Medical Model Performance under Distributed Deployment

    Sep 29, 2026Yifei Wang, Xiaohan Zhang, Youtao Ding +4Multimodal Model EvaluationDistributed Inference

    Shanghai Jiao Tong University, China · Beihang University, China · Nanyang Technological University, Singapore +1

  8. Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines

    Sep 28, 2026Murtaza Rangwala, Richard O. Sinnott, Rajkumar BuyyaLLM InferenceLLM Inference Scheduling

    Quantum Cloud Computing and Distributed Systems (qCLOUDS) Lab, School of Computing and Information Systems, The University of Melbourne, Australia

  9. GenomeOcean Anywhere: Private WebGPU Inference for Genome MoEs

    Sep 27, 2026Guang Yang, Fengchen LiuGenomic Foundation ModelsDistributed Inference

    University of California, Los Angeles · University of California, Berkeley

  10. When Does Communication Help? Beyond Spectral Descriptions of Collective Intelligence

    Sep 20, 2026Xuening WuShared Decision-MakingMulti-Agent Communication

    AI Engineering, Pfizer, Shanghai, 200041

  11. vidax: A Unified JAX Framework for Video Generative Models on Accelerator Meshes

    Sep 16, 2026Congyue DengVideo Diffusion ModelsTensor Parallelism

    Massachusetts Institute of Technology

  12. End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services

    Sep 15, 2026Zhen Li, Jun Cai, Haoran Gao +2LLM Inference SchedulingLLM Request Scheduling

    Department of Electrical and Computer Engineering, Concordia University, Montreal, QC, H3G 1M8, Canada · Department of Computer Science, Hang Seng University of Hong Kong, Hong Kong SAR

  13. Recovering Governing Dynamics from Distributed Observations via Exact Spline Merging

    Sep 15, 2026Naveen MysorePDE DiscoveryDistributed Inference

    University of California, Santa Barbara · Dyssonance AI

  14. MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks

    Sep 13, 2026Sokratis Nikolaidis, Stylianos I. Venieris, Leonidas Malachias +1Edge InferenceDistributed Inference

    National Technical University of Athens, Athens, Greece · Samsung AI Center, Cambridge, UK

  15. mKernel: Fast Multi-GPU, Multi-Node Fused Kernels

    Sep 11, 2026Ziming Mao, Yihan Zhang, Shawn Wei Chew +5GPU Kernel OptimizationCommunication-Efficient Distributed Training

    UC Berkeley · UC Davis · UCLA +1

  16. Composable CXL Memory as a Kubernetes-Native Shared Memory for LLM Serving

    Sep 9, 2026Hongjian Fan, Kevin Zhang, David Habinsky +1LLM Inference EfficiencyKV Caching

    Research Group, Seagate Technology, LLC

  17. A Fundamental Limit in Decentralized Decision-Making

    Sep 7, 2026Marco Carpentiero, Felice Scala, Vincenzo Matta +1ClassificationDistributed Inference

    Department of Information and Electrical Engineering and Applied Mathematics (DIEM), University of Salerno, I-84084 Fisciano (SA), Italy · Institute of Electrical and Micro Engineering, EPFL, CH-1015 Lausanne, Switzerland

  18. Manifold-Aware General Coded Computing for Straggler-Resilient Distributed Computing

    Sep 1, 2026Parsa Moradi, Mohammad Ali Maddah-AliDistributed Inference

    University of Minnesota, Twin Cities Minneapolis, MN, USA

  19. Decoupled I/O-Dominant Pipelines for Large-Scale Whole-Slide Image Embedding Extraction

    Aug 27, 2026Mayanka Chandrashekar, Xi Zhang, Ethan Seefried +3Whole-Slide Image AnalysisDistributed Inference

    Oak Ridge National Laboratory

  20. User-Assisted Collaborative Distributed Inference for Efficient QoS-Aware Autoscaling

    Aug 12, 2026Alfreds Lapkovskis, Ali Beikmohammadi, Sindri Magnússon +1Distributed InferenceOnline Resource Allocation

    Department of Computer Systems and Sciences, Stockholm University SE-106 91 Stockholm, Sweden

  21. HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference

    Aug 1, 2026Xin Yuan, Ning Li, Wenchao Xu +2Edge InferenceCost-Aware Inference

    School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China · School of Artificial Intelligence, University of Science and Technology Beijing, China · Division of Integrative Systems and Design, Hong Kong University of Science and Technology, Hong Kong +1

  22. Denial of Deadline: Network-Driven Accuracy Collapse in Distributed Inference Pipelines

    Jul 27, 2026Jhonatan Tavori, Gur-Eyal Sela, Ion Stoica +1Adversarial AttacksEdge Computing

    Columbia University · UC Berkeley

  23. Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs

    Jul 25, 2026Zhihao Xu, Hao Zhong, Zeting Zhou +9Heterogeneous ComputingDistributed Inference

    Shanghai Jiao Tong University · Lenovo Information Products (Shenzhen) Ltd · Lenovo (Beijing) Ltd

  24. Integrity of peer-to-peer distributed LLM inference under malicious nodes

    Jul 21, 2026Mert Cihangiroglu, Antonino NoceraLLM SecurityDistributed Inference

    DCALab, Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Italy

  25. DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse

    Jul 17, 2026Yuyang Chen, Runxin Zhong, Zan Zong +3Diffusion TransformerEfficient Attention

    Shanghai Jiao Tong University, Shanghai AI Laboratory Shanghai, China · Tsinghua University Beijing, China · University of Science and Technology Beijing Beijing, China +1

  26. Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs

    Jul 13, 2026Weijia Han, Lisha QuLLM QuantizationGPU Kernel Optimization

    University of Washington

  27. Stateful Worlds, Stateless Elasticity: Exact-State Serving for Interactive World Models

    Jul 11, 2026Jin Li, Jiawei ChenDistributed InferenceOnline Resource Allocation

    Harvard University · Harvard University Cambridge, MA, USA · Independent Researcher

  28. D-CLIPSE: Distributed Consensus-based Localization with Passive Listening on Shared State Exchange

    Jul 9, 2026Kyle Biron-Gricken, James Richard ForbesCooperative LocalizationRobot State Estimation

  29. CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems

    Jul 8, 2026Tingkai Liu, Muralidhar Andoorveedu, Sanjoy Das +2GPU Kernel OptimizationHigh-Performance Computing

    University of Illinois Urbana-Champaign · NVIDIA Corporation

  30. Design-CP: Context Parallelism for Design of Protein Nanoparticles

    Jul 3, 2026Lorenzo Tarricone, Helen E. Eisenach, Aiko Muraishi +1GPU AccelerationHigh-Performance Computing

    Department of Statistics, University of Oxford, Oxford, UK · Ellison Institute of Technology Oxford, Oxford, UK · Institute for Protein Design, University of Washington, Seattle, USA +2