Distributed Inference

Ask about this topic

Overview

Distributed inference studies how to divide or coordinate model execution across GPUs, edge devices, robots, sensors, and cloud services when one machine is insufficient or unsuitable. Common approaches include pipeline and tensor parallelism, mixture-of-experts placement, edge–cloud partitioning, federated or collaborative prediction, and communication-aware scheduling of requests and cached state. Systems are increasingly designed for heterogeneous hardware and changing network conditions, with simulation and real-cluster benchmarks used to assess latency, throughput, accuracy, and resource use. The central tension is that distributing computation can add communication, coordination, and reliability costs, so balancing speed and scale against privacy, consistency, and robustness remains an open challenge.

Latest papers

CardsList

75 papers8 papers in the last four weeks, down 20% on the four weeks before. 0.0% of all new papers.

  1. Spark Policy Toolkit: Semantic Contracts and Scalable Execution for Policy Learning in Spark

    Apr 27, 2026Zeyu BaiEfficient InferencePolicy Learning

  2. Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns

    Apr 25, 2026Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6Expert Load BalancingMixture-of-Experts Inference

    1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.

  3. Time, Causality, and Observability Failures in Distributed AI Inference Systems

    Apr 23, 2026Ankur Sharma, Deep Shah, David Lariviere +1Temporal ConsistencyDistributed Inference

    Unified Intelligent Infrastructure workstream at the Open Compute Project (OCP)

  4. NeuroMesh: A Unified Neural Inference Framework for Decentralized Multi-Robot Collaboration

    Apr 16, 2026Yang Zhou, Yash Shetye, Long Quang +8Efficient Neural Network InferenceMulti-Robot Systems

    New York University, New York, USA · U.S. Army Combat Capabilities Development Command, Army Research Laboratory, Adelphi, MD 20783, USA · Vanderbilt University, Tennessee, USA +1

  5. On-Orbit Space AI: Federated, Multi-Agent, and Collaborative Algorithms for Satellite Constellations

    Apr 15, 2026Ziyang WangMulti-Agent CoordinationDistributed Inference

  6. Tight Convergence Rates for Online Distributed Linear Estimation with Adversarial Measurements

    Apr 7, 2026Nibedita Roy, Vishal Halder, Gugan Thoppe +4Distributed InferenceRobust Estimation

    Dept. of Computer Science and Automation, Indian Institute of Science, Bengaluru 560012, India · Department of Computer Science, IMT Atlantique, Plouzané 29280, France

  7. DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72

    Apr 2, 2026Wanqian Li, Jintao Peng, Zongfei Jing +7LLM Inference AccelerationDistributed Inference

    NVIDIA

  8. Trust-Aware Routing for Distributed Generative AI Inference at the Edge

    Mar 30, 2026Chanh Nguyen, Erik ElmrothDistributed InferenceEdge Computing

    Department of Computing Science, Ume˚a University, SE-90187, Sweden

  9. UNIFERENCE: A Discrete Event Simulation Framework for Developing Distributed AI Models

    Mar 27, 2026Doğaç Eldenk, Stephen XiaDistributed Inference

    Department of Computer Science, Northwestern University, Evanston, United States · Department of Computer Engineering, Northwestern University, Evanston, United States

  10. RoboECC: Multi-Factor-Aware Edge-Cloud Collaborative Deployment for VLA Models

    Mar 21, 2026Zihao Zheng, Hangyu Cao, Jiayu Chen +6Edge-Cloud Collaborative InferenceEfficient VLA Model Inference

    School of Computer Science, Peking University, Beijing, China · School of Computer Science, South China University of Technology, Guangzhou, China · School of Artificial Intelligence, Beijing Normal University, Beijing, China +1

  11. Joint Optimization of Model Partitioning and Resource Allocation for Anti-Jamming Collaborative Inference Systems

    Mar 3, 2026Mengru Wu, Jiawei Li, Jiaqi Wei +3Distributed InferenceEdge Computing

    College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China · School of Communications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing 210003, China · Department of Electronic and Electrical Engineering, University College London, WC1E 7JE London, U.K. +2

  12. A Framework for Hybrid Collective Inference in Distributed Sensor Networks

    Feb 19, 2026Andrew Nash, Dirk Pesch, Krishnendu GuhaDistributed InferenceMulti-Agent Reasoning

    School of Computer Science and IT, University College Cork, Ireland.

  13. Cloud Is Closer Than It Appears: Revisiting the Tradeoffs of Distributed Real-Time Inference

    Feb 17, 2026Pragya Sharma, Hang Qiu, Mani SrivastavaDistributed InferenceCyber-Physical Systems

    University of California Los Angeles · University of California Riverside

  14. R2CC: Reliable and Resilient Collective Communication for Distributed Learning

    Dec 31, 2025Wei Wang, Nengneng Yu, Sixian Xiong +1Fault-Tolerant Distributed TrainingDistributed Inference

    University of Maryland, College Park

  15. Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

    Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model InferenceLLM Inference

    MBZUAI · UESTC · City University of Hong Kong +1