Distributed Inference

Ask about this topic

Overview

Distributed inference studies how to divide or coordinate model execution across GPUs, edge devices, robots, sensors, and cloud services when one machine is insufficient or unsuitable. Common approaches include pipeline and tensor parallelism, mixture-of-experts placement, edge–cloud partitioning, federated or collaborative prediction, and communication-aware scheduling of requests and cached state. Systems are increasingly designed for heterogeneous hardware and changing network conditions, with simulation and real-cluster benchmarks used to assess latency, throughput, accuracy, and resource use. The central tension is that distributing computation can add communication, coordination, and reliability costs, so balancing speed and scale against privacy, consistency, and robustness remains an open challenge.

Latest papers

CardsList

75 papers8 papers in the last four weeks, down 20% on the four weeks before. 0.0% of all new papers.

Apr 27, 2026

Spark Policy Toolkit: Semantic Contracts and Scalable Execution for Policy Learning in Spark

Custom policy-learning pipelines in Spark fail for two coupled systems reasons: rowwise Python execution makes inference impractical, and driver-side candidate materialization makes split search fragile at feature scale. We present Spark Policy Toolkit, a semantics-governed systems toolkit for scalable policy learning in Spark. The toolkit provides two Spark-native primitives: partition-initialized vectorized inference through mapInPandas and mapInArrow, and collect-less split search that scores candidates on executors. Both primitives are governed by one fixed-input semantic contract: the same rows, feature order, treatment vocabulary, preprocessing manifest, and split boundaries must preserve per-row score vectors, best-split decisions, and end-to-end learned policy outputs. The evaluation combines practical baseline ladders, backend parity checks, measured split-search scale results, synthetic and Hillstrom end-to-end policy preservation, missingness stress, partition and order perturbation tests, quantile-boundary sensitivity, and a concrete adversarial failure catalog. On a 40-worker Databricks cluster, mapInArrow reaches 4.72M rows/s at 10M matched rows and 7.23M rows/s at 50M rows, while collect-less split search remains valid from F = 10 through F = 1000 with 124000 candidate rows, where the driver-collect baseline is intentionally skipped. Across 24 backend-ablation settings, mapInArrow wins 18 while mapInPandas wins 6, so the paper treats backend choice as workload-dependent rather than universal. Once the fixed-input lock is enforced, all six tested repartition/coalesce/shuffle perturbations preserve identical signatures; before lock, all six drift. The central result is not speed alone: throughput and collect-less execution are the mechanisms that let policy semantics survive at Spark scale.

Zeyu Bai

Efficient InferencePolicy LearningDistributed Inference
AskPDF
Apr 25, 2026

Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns

Most recent state-of-the-art (SOTA) large language models (LLMs) use Mixture-of-Experts (MoE) architectures to scale model capacity without proportional per-token compute, enabling higher-quality outputs at manageable serving costs. However, MoE inference at scale is fundamentally bottlenecked by expert load imbalance and inefficient token routing, especially in multi-node deployments where tokens are not guaranteed to be routed to local experts, resulting in significant inter-node all-to-all communication overhead. To systematically characterize these challenges, we profile SOTA open-source MoE models, including Llama 4 Maverick, DeepSeek V3-671B, and Qwen3-230B-A22B, on various datasets and collected over 100k real expert activation traces. Upon studying the expert activation patterns, we uncover various persistent properties across all the frontier MoE models: variable expert load imbalance, domain-specific expert activation where expert popularity shifts across task families (code, math, chat, general), and a strong correlation between prefill and decode expert activations. Motivated by these findings, we propose workload-aware micro-batch grouping and an expert placement strategy to maximize token locality to the destination expert, thereby reducing inter-node communication. Across models and datasets, these optimizations help reduce all2all communication data up to 20, resulting in lower MoE decode latency and better accelerator utilization.

Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6

1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.

Expert Load BalancingMixture-of-Experts InferenceLLM RoutingMixture-of-Experts ModelsDistributed InferenceCommunication-Efficient Mixture-of-Experts
AskPDF
Apr 23, 2026

Time, Causality, and Observability Failures in Distributed AI Inference Systems

Distributed AI inference pipelines rely heavily on timestamp-based observability to understand system behavior. This work demonstrates that even small clock skew between nodes can cause observability to become causally incorrect while the system itself remains functionally correct and performant. We present controlled experiments on a multi-node AI inference pipeline, where clock skew is introduced at a single stage. Results show that no violations are observed under synchronized conditions and up to 3 ms skew, while clear causality violations emerge by 5 ms. Despite this, system throughput and output correctness remain largely unaffected. We further observe that violation behavior is not strictly static. In longer runs, negative span rates may stabilize or decrease over time, indicating that effective skew evolves due to relative clock drift between nodes. Experiments were conducted using Kafka and ZeroMQ transports, with consistent results across both. Aeron is under active exploration but is not yet included in the completed validation set. These findings suggest that observability correctness depends not only on system functionality but also on precise time alignment, and that timing must be treated as a first-class concern in distributed AI systems.

Ankur Sharma, Deep Shah, David Lariviere +1

Unified Intelligent Infrastructure workstream at the Open Compute Project (OCP)

Temporal ConsistencyDistributed Inference
AskPDF
Apr 16, 2026

NeuroMesh: A Unified Neural Inference Framework for Decentralized Multi-Robot Collaboration

Deploying learned multi-robot models on heterogeneous robots remains challenging due to hardware heterogeneity, communication constraints, and the lack of a unified execution stack. This paper presents NeuroMesh, a multi-domain, cross-platform, and modular decentralized neural inference framework that standardizes observation encoding, message passing, aggregation, and task decoding in a unified pipeline. NeuroMesh combines a dual-aggregation paradigm for reduction- and broadcast-based information fusion with a parallelized architecture that decouples cycle time from end-to-end latency. Our high-performance C++ implementation leverages Zenoh for inter-robot communication and supports hybrid GPU/CPU inference. We validate NeuroMesh on a heterogeneous team of aerial and ground robots across collaborative perception, decentralized control, and task assignment, demonstrating robust operation across diverse task structures and payload sizes. We plan to release NeuroMesh as an open-source framework to the community.

Yang Zhou, Yash Shetye, Long Quang +8

New York University, New York, USA · U.S. Army Combat Capabilities Development Command, Army Research Laboratory, Adelphi, MD 20783, USA · Vanderbilt University, Tennessee, USA +1

Efficient Neural Network InferenceMulti-Robot SystemsMulti-Robot CoordinationMulti-Agent CommunicationDistributed Inference
AskPDF
Apr 15, 2026

On-Orbit Space AI: Federated, Multi-Agent, and Collaborative Algorithms for Satellite Constellations

Satellite constellations are transforming space systems from isolated spacecraft into networked, software-defined platforms capable of on-orbit perception, decision making, and adaptation. Yet much of the existing AI studies remains centered on single-satellite inference, while constellation-scale autonomy introduces fundamentally new algorithmic requirements: learning and coordination under dynamic inter-satellite connectivity, strict SWaP-C limits, radiation-induced faults, non-IID data, concept drift, and safety-critical operational constraints. This survey consolidates the emerging field of on-orbit space AI through three complementary paradigms: (i) {federated learning} for cross-satellite training, personalization, and secure aggregation; (ii) {multi-agent algorithms} for cooperative planning, resource allocation, scheduling, formation control, and collision avoidance; and (iii) {collaborative sensing and distributed inference} for multi-satellite fusion, tracking, split/early-exit inference, and cross-layer co-design with constellation networking. We provide a system-level view and a taxonomy that unifies collaboration architectures, temporal mechanisms, and trust models. To support community development and keep this review actionable over time, we continuously curate relevant papers and resources at https://github.com/ziyangwang007/AI4Space.

Ziyang Wang

Multi-Agent CoordinationDistributed InferenceFederated LearningCooperative Perception
AskPDF
Apr 7, 2026

Tight Convergence Rates for Online Distributed Linear Estimation with Adversarial Measurements

We study mean estimation of a random vector XX in a distributed parameter-server-worker setup. Worker ii observes samples of ai⊤Xa_i^\top X, where ai⊤a_i^\top is the iith row of a known sensing matrix AA. The key challenges are adversarial measurements and asynchrony: a fixed subset of workers may transmit corrupted measurements, and workers are activated asynchronously--only one is active at any time. In our previous work, we proposed a two-timescale ℓ1\ell_1-minimization algorithm and established asymptotic recovery under a null-space-property-like condition on AA. In this work, we establish tight non-asymptotic convergence rates under the same null-space-property-like condition. We also identify relaxed conditions on AA under which exact recovery may fail but recovery of a projected component of E[X]\mathbb{E}[X] remains possible. Overall, our results provide a unified finite-time characterization of robustness, identifiability, and statistical efficiency in distributed linear estimation with adversarial workers, with implications for network tomography and related distributed sensing problems.

Nibedita Roy, Vishal Halder, Gugan Thoppe +4

Dept. of Computer Science and Automation, Indian Institute of Science, Bengaluru 560012, India · Department of Computer Science, IMT Atlantique, Plouzané 29280, France

Distributed InferenceRobust EstimationDistributed Optimization
AskPDF
Apr 2, 2026

DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72

Large language model (LLM) inference increasingly depends on multi-GPU execution, yet existing inference parallelization strategies require layer-wise inter-rank synchronization, making end-to-end performance sensitive to workload imbalance. We present DWDP (Distributed Weight Data Parallelism), an inference parallelization strategy that preserves data-parallel execution while offloading MoE weights across peer GPUs and fetching missing experts on demand. By removing collective inter-rank synchronization, DWDP allows each GPU to progress independently. We further address the practical overheads of this design with two optimizations for split-weight management and asynchronous remote-weight prefetch. Implemented in TensorRT-LLM and evaluated with DeepSeek-R1 on GB200 NVL72, DWDP improves end-to-end output TPS/GPU by 8.8% at comparable TPS/user in the 20-100 TPS/user serving range under 8K input sequence length and 1K output sequence length.

Wanqian Li, Jintao Peng, Zongfei Jing +7

NVIDIA

LLM Inference AccelerationDistributed InferenceMixture-of-Experts Inference
AskPDF
Mar 30, 2026

Trust-Aware Routing for Distributed Generative AI Inference at the Edge

Emerging deployments of Generative AI increasingly execute inference across decentralized and heterogeneous edge devices rather than on a single trusted server. In such environments, a single device failure or misbehavior can disrupt the entire inference process, making traditional best-effort peer-to-peer routing insufficient. Coordinating distributed generative inference therefore requires mechanisms that explicitly account for reliability, performance variability, and trust among participating peers. In this paper, we present G-TRAC, a trust-aware coordination framework that integrates algorithmic path selection with system-level protocol design to ensure robust distributed inference. First, we formulate the routing problem as a Risk-Bounded Shortest Path computation and introduce a polynomial-time solution that combines trust-floor pruning with Dijkstra's search, achieving sub-millisecond median routing latency at practical edge scales, and remaining below 10 ms at larger scales. Second, to operationally support the routing logic in dynamic environments, the framework employs a Hybrid Trust Architecture that maintains global reputation state at stable anchors while disseminating lightweight updates to edge peers via background synchronization. Experimental evaluation on a heterogeneous testbed of commodity devices demonstrates that G-TRAC significantly improves inference completion rates, effectively isolates unreliable peers, and sustains robust execution even under node failures and network partitions.

Chanh Nguyen, Erik Elmroth

Department of Computing Science, Ume˚a University, SE-90187, Sweden

Distributed InferenceEdge Computing
AskPDF
Mar 27, 2026

UNIFERENCE: A Discrete Event Simulation Framework for Developing Distributed AI Models

Developing and evaluating distributed inference algorithms remains difficult due to the lack of standardized tools for modeling heterogeneous devices and networks. Existing studies often rely on ad-hoc testbeds or proprietary infrastructure, making results hard to reproduce and limiting exploration of hypothetical hardware or network configurations. We present UNIFERENCE, a discrete-event simulation (DES) framework designed for developing, benchmarking, and deploying distributed AI models within a unified environment. UNIFERENCE models device and network behavior through lightweight logical processes that synchronize only on communication primitives, eliminating rollbacks while preserving the causal order. It integrates seamlessly with PyTorch Distributed, enabling the same codebase to transition from simulation to real deployment. Our evaluation demonstrates that UNIFERENCE profiles runtime with up to 98.6% accuracy compared to real physical deployments across diverse backends and hardware setups. By bridging simulation and deployment, UNIFERENCE provides an accessible, reproducible platform for studying distributed inference algorithms and exploring future system designs, from high-performance clusters to edge-scale devices. The framework is open-sourced at https://github.com/Dogacel/Uniference.

Doğaç Eldenk, Stephen Xia

Department of Computer Science, Northwestern University, Evanston, United States · Department of Computer Engineering, Northwestern University, Evanston, United States

Distributed Inference
AskPDF
Mar 21, 2026

RoboECC: Multi-Factor-Aware Edge-Cloud Collaborative Deployment for VLA Models

Vision-Language-Action (VLA) models are mainstream in embodied intelligence but face high inference costs. Edge-Cloud Collaborative (ECC) deployment offers an effective fix by easing edge-device computing pressure to meet real-time needs. However, existing ECC frameworks are suboptimal for VLA models due to two challenges: (1) Diverse model structures hinder optimal ECC segmentation point identification; (2) Even if the optimal split point is determined, changes in network bandwidth can cause performance drift. To address these issues, we propose a novel ECC deployment framework for various VLA models, termed RoboECC. Specifically, we propose a model-hardware co-aware segmentation strategy to help find the optimal segmentation point for various VLA models. Moreover, we propose a network-aware deployment adjustment approach to adapt to the network fluctuations for maintaining optimal performance. Experiments demonstrate that RoboECC achieves a speedup of up to 3.28x with only 2.55%~2.62% overhead.

Zihao Zheng, Hangyu Cao, Jiayu Chen +6

School of Computer Science, Peking University, Beijing, China · School of Computer Science, South China University of Technology, Guangzhou, China · School of Artificial Intelligence, Beijing Normal University, Beijing, China +1

Edge-Cloud Collaborative InferenceEfficient VLA Model InferenceEdge-Cloud ComputingEdge ComputingDistributed InferenceVLMs for Robotics
AskPDF
Mar 3, 2026

Joint Optimization of Model Partitioning and Resource Allocation for Anti-Jamming Collaborative Inference Systems

With the increasing computational demands of deep neural network (DNN) inference on resource-constrained devices, DNN partitioning-based device-edge collaborative inference has emerged as a promising paradigm. However, the transmission of intermediate feature data is vulnerable to malicious jamming, which significantly degrades the overall inference performance. To counter this threat, this letter focuses on an anti-jamming collaborative inference system in the presence of a malicious jammer. In this system, a DNN model is partitioned into two distinct segments, which are executed by wireless devices and edge servers, respectively. We first analyze the effects of jamming and DNN partitioning on inference accuracy via data regression. Based on this, our objective is to maximize the system's revenue of delay and accuracy (RDA) under inference accuracy and computing resource constraints by jointly optimizing computation resource allocation, devices' transmit power, and DNN partitioning. To address the mixed-integer nonlinear programming problem, we propose an efficient alternating optimization-based algorithm, which decomposes the problem into three subproblems that are solved via Karush-Kuhn-Tucker conditions, convex optimization methods, and a quantum genetic algorithm, respectively. Extensive simulations demonstrate that our proposed scheme outperforms baselines in terms of RDA.

Mengru Wu, Jiawei Li, Jiaqi Wei +3

College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China · School of Communications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing 210003, China · Department of Electronic and Electrical Engineering, University College London, WC1E 7JE London, U.K. +2

Distributed InferenceEdge ComputingWireless Resource AllocationWireless Communications
AskPDF
Feb 19, 2026

A Framework for Hybrid Collective Inference in Distributed Sensor Networks

With the ever-increasing range of applications of Internet in Things (IoT) and sensor networks, challenges are emerging in various categories of classification tasks. Applications such as vehicular networking, UAV swarm coordination and cyber-physical systems require global classification over distributed sensors, with tight constraints on communication and computation resources. There has been much research in decentralized and distributed data-exchange for communication-efficient collective inference. Likewise, there has been considerable research involving the use of cloud and edge computing paradigms for efficient task allocation. To the best of our knowledge, there has been no research on the integration of these two concepts to create a hybrid cloud and distributed approach that makes dynamic runtime communication strategy decisions. In this paper, we focus on aspects of combining distributed and hierarchical communication and classification approaches for collective inference. We derive optimal policies for agents that implement this hybrid approach, and evaluate their performance under various scenarios of the distribution of underlying data. Our analysis shows that this approach can maintain a high level of classification accuracy (comparable to that of centralised joint inference over all data), at reduced theoretical communication cost. We expect there is potential for our approach to facilitate efficient collective inference for real-world applications, including instances that involves more complex underlying data distributions.

Andrew Nash, Dirk Pesch, Krishnendu Guha

School of Computer Science and IT, University College Cork, Ireland.

Distributed InferenceMulti-Agent ReasoningInternet of Things
AskPDF
Feb 17, 2026

Cloud Is Closer Than It Appears: Revisiting the Tradeoffs of Distributed Real-Time Inference

The increasing deployment of deep neural networks (DNNs) in cyber-physical systems (CPS) enhances perception fidelity, but imposes substantial computational demands on execution platforms, posing challenges to real-time control deadlines. Traditional distributed CPS architectures typically favor on-device inference to avoid network variability and contention-induced delays on remote platforms. However, this design choice places significant energy and computational demands on the local hardware. In this work, we revisit the assumption that cloud-based inference is intrinsically unsuitable for latency-sensitive control tasks. We demonstrate that, when provisioned with high-throughput compute resources, cloud platforms can effectively amortize network and queueing delays, enabling them to match or surpass on-device performance for real-time decision-making. Specifically, we develop a formal analytical model that characterizes distributed inference latency as a function of the sensing frequency, platform throughput, network delay, and task-specific safety constraints. We instantiate this model in the context of emergency braking for autonomous driving and validate it through extensive simulations using real-time vehicular dynamics. Our empirical results identify concrete conditions under which cloud-based inference adheres to safety margins more reliably than its on-device counterpart. These findings challenge prevailing design strategies and suggest that the cloud is not merely a feasible option, but often the preferred inference location for distributed CPS architectures. In this light, the cloud is not as distant as traditionally perceived; in fact, it is closer than it appears.

Pragya Sharma, Hang Qiu, Mani Srivastava

University of California Los Angeles · University of California Riverside

Distributed InferenceCyber-Physical SystemsEdge-Cloud Computing
AskPDF
Dec 31, 2025

R2CC: Reliable and Resilient Collective Communication for Distributed Learning

Modern ML training and inference now span hundreds to tens of thousands of GPUs, where failures can waste 10-15% of GPU hours due to slow recovery. Common network errors and link fluctuations can cause up to 50% of job interruptions, triggering timeouts that often terminate entire jobs, forcing expensive checkpoint rollback during training and request reprocessing during inference. We present R2CC, a fault-tolerant communication library that provides lossless, low-overhead failover by exploiting multi-NIC hardware. R2CC performs rapid connection migration, bandwidth-aware load redistribution, and resilient collective algorithms to maintain progress under failures. We evaluate R2CC on four 8-GPU H100 InfiniBand servers and via large-scale ML simulators modeling up to 1024 GPUs with diverse failure patterns. Experiments show that R2CC is highly robust to inter-node network failures, incurring less than 1.1% training and 3% inference overhead under active failure scenarios. Compared to existing fault-tolerant approaches, R2CC reduces failure-induced overhead by up to 92% for training and 98% for inference. R2CC is open-sourced at https://github.com/r2cc-project/R2CC.

Wei Wang, Nengneng Yu, Sixian Xiong +1

University of Maryland, College Park

Fault-Tolerant Distributed TrainingDistributed Inference
AskPDF
Apr 7, 2025

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

On-device inference offers privacy, offline use, and instant response, but consumer hardware restricts large language models (LLMs) to low throughput and capability. To overcome this challenge, we present prima.cpp, a distributed on-device inference system that runs 30-70B LLMs on consumer home clusters with mixed CPUs/GPUs, insufficient RAM/VRAM, slow disks, Wi-Fi links, and heterogeneous OSs. We introduce pipelined-ring parallelism (PRP) to overlap disk I/O with compute and communication, and address the prefetch-release conflict in mmap-based offloading. We further propose Halda, a heterogeneity-aware scheduler that co-optimizes per-device CPU/GPU workloads and device selection under RAM/VRAM constraints. On four consumer home devices, a 70B model reaches 674 ms/token TPOT with <6% memory pressure, and a 32B model with speculative decoding achieves 26 tokens/s. Compared with llama.cpp, exo, and dllama, our proposed prima.cpp achieves 5-17 times lower TPOT, supports fine-grained model sizes from 8B to 70B, ensures broader cross-OS and quantization compatibility, and remains OOM-free, while also being Wi-Fi tolerant, privacy-preserving, and hardware-independent. The code is available at https://github.com/OpenCPIL/prima.cpp.

Zonghang Li, Tao Li, Wenjiao Feng +8

MBZUAI · UESTC · City University of Hong Kong +1

On-Device Language Model InferenceLLM InferenceLLM Inference SchedulingLLM Inference AccelerationPipeline ParallelismDistributed Inference
AskPDF