Distributed Inference

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 55

All topics

UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
Jun 2, 2026Xinming Wei, Chao Jin, Tuo Dai +10Expert Parallelism Expert Load Balancing
School of Computer Science, Peking University · Xiaohongshu Inc. · Shanghai AI Laboratory +1
NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
Jun 2, 2026Mubarak Adetunji OjewaleLLM Inference KV Caching
Cloud Competency Centre, National College of Ireland
CADET: A Modular Platform for Evaluating Distributed Cooperative Autonomy in Connected Autonomous Vehicles
Jun 2, 2026Pragya Sharma, Brian Wang, Mani SrivastavaEdge Computing Autonomous Driving
University of California Los Angeles
Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics
May 31, 2026Bole Ma, Jan Eitzinger, Harald Köstler +1LLM Inference KV Caching
Beyond Task-Agnostic: Task-Aware Grouping for Communication-Efficient Multi-Task MoE Inference
May 31, 2026Zhiyao Xu, Aoxue Liu, Zhanjie Ding +3Mixture-of-Experts Inference Mixture-of-Experts Models
Tsinghua Shenzhen International Graduate School, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China
Lodestar: An Online-Learning LLM Inference Router
May 31, 2026Gangmuk Lim, Wanyu Zhao, Brighten Godfrey +3Inference-Time Optimization LLM Inference Scheduling
UIUC · Bytedance · University of Edinburgh
How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving
May 27, 2026Hanjiang Wu, Abhimanyu Rajeshkumar Bambhaniya, Sarbartha Banerjee +9LLM Serving LLM Inference
1Georgia Institute of Technology · 5Infravana · 2Intel +2
Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization
May 22, 2026Alexander Long, Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan +5LLM Inference Decentralized Learning
Pluralis Research
Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
May 20, 2026Yicheng Feng, Xin Tan, Yangtao Deng +3LLM Serving Disaggregated LLM Serving
The Chinese University of Hong Kong · Anuttacon · StepFun
Federated Martingale Posterior Samping
May 18, 2026Boning Zhang, Matteo Zecchin, Mingzhao Guo +2Bayesian Neural Networks Posterior Sampling
School of Computing, University of Glasgow, Glasgow G12 8QQ, U.K. · Communication Systems Department, EURECOM, 06904 Sophia Antipolis, France · Institute for Intelligent Networked Systems, Northeastern University London, London E1 8PH, U.K.
Byzantine-Robust Distributed Sparse Learning Revisited
May 13, 2026Yuxuan Wang, Lixin Zhang, Kangqiang LiRobust Regression Distributed Optimization
School of Mathematical Sciences, Zhejiang University, Hangzhou, 310027, Zhejiang, China · School of Statistics and Mathematics, Zhejiang Gongshang University, Hangzhou, 310018, Zhejiang, China · Information Center, Hubei Provincial Tobacco Monopoly Administration, Wuhan, 430030, Hubei, China
ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inference
May 11, 2026Han Meng, Danny Willow Liu, Dong LiDiffusion Transformer GPU Acceleration
University of California, Merced · University of Chicago · University of California, Merced and Yotta Labs +1
ShardTensor: Domain Parallelism for Scientific Machine Learning
May 11, 2026Corey Adams, Peter Harrington, Akshay Subramaniam +4Scientific ML Distributed Training
NVIDIA · Santa Clara, CA, USA
PoHAR: Understanding Hyperlocal Human Activities with Pollution Sensor Networks
May 10, 2026Prasenjit Karmakar, Karthik Reddy, Sandip ChakrabortyHuman Activity Recognition Internet of Things
Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, India.
Towards Distributed Inference of LLMs on a P2P Network
May 7, 2026Shabari S Nair, Krishanu SainiLLM Inference KV Caching
Department of Computer Science, The University of Texas at Austin.
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
May 7, 2026Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini +1Expert Parallelism Mixture-of-Experts Inference
Stanford University
Enabling Federated Inference via Unsupervised Consensus Embedding
May 7, 2026Yui Hashimoto, Takayuki Nishio, Yuichi Kitagawa +1Ensemble Learning Privacy-Preserving ML
School of Engineering, Institute of Science Tokyo, Tokyo 152-8550, Japan · Research & Development Group, Hitachi Ltd., Tokyo 185-8601, Japan
MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
May 3, 2026Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan +5Expert Parallelism LLM Serving
University of Virginia · 2Snowflake AI Research · 3Harvard University
SpaceMoE: Realizing Distributed Mixture-of-Experts Inference over Space Networks
May 1, 2026Zhanwei Wang, Huiling Yang, Min Sheng +2Mixture-of-Experts Language Models Mixture-of-Experts Inference
Department of Electrical and Computer Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China · State Key Laboratory of Integrated Service Networks, Institute of Information Science, Xidian University (XDU), Xi’an, Shaanxi, China · Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology (HKUST), Hong Kong SAR, China
AI Inference as Relocatable Electricity Demand: A Latency-Constrained Energy-Geography Framework
Apr 30, 2026Xubin Luo, Cheng YangEnergy-Efficient ML Cost-Aware Inference
School of Computer Science and Artificial Intelligence, Southwestern University of Finance and Economics, Chengdu, China
Spark Policy Toolkit: Semantic Contracts and Scalable Execution for Policy Learning in Spark
Apr 27, 2026Zeyu BaiEfficient Inference Policy Learning
Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
Apr 25, 2026Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6Expert Load Balancing Mixture-of-Experts Inference
1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.
Time, Causality, and Observability Failures in Distributed AI Inference Systems
Apr 23, 2026Ankur Sharma, Deep Shah, David Lariviere +1Temporal Consistency Distributed Inference
Unified Intelligent Infrastructure workstream at the Open Compute Project (OCP)
NeuroMesh: A Unified Neural Inference Framework for Decentralized Multi-Robot Collaboration
Apr 16, 2026Yang Zhou, Yash Shetye, Long Quang +8Efficient Neural Network Inference Multi-Robot Systems
New York University, New York, USA · U.S. Army Combat Capabilities Development Command, Army Research Laboratory, Adelphi, MD 20783, USA · Vanderbilt University, Tennessee, USA +1
Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model Inference LLM Inference
MBZUAI · UESTC · City University of Hong Kong +1

Jun 2, 2026, cs.DCJ/K move · Enter open · S save

UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing

Xinming Wei, Chao Jin, Tuo Dai, Yinmin Zhong+9

School of Computer Science, Peking University · Xiaohongshu Inc. · Shanghai AI Laboratory+1

Expert Parallelism Expert Load Balancing High-Performance Computing Mixture-of-Experts Inference Distributed Training Mixture-of-Experts Models+1

Open paper AskPDF arXiv

Large-scale expert parallelism (EP) is becoming pivotal for training and serving frontier MoE models, but it also amplifies device-level expert load imbalance into compute stragglers, token all-to-all bottlenecks, and activation-memory spikes. Existing balancers redistribute experts periodically based on historical load, which becomes unreliable for production deployments with non-stationary load patterns. We present UltraEP, the first exact-load, real-time balancer for large-EP MoE training and serving prefill on rack-scale nodes (RSNs). Leveraging the extended scale-up connectivity among dozens of GPUs within RSNs, UltraEP rebalances every microbatch and layer on critical paths, which requires nontrivial co-design of plan solving and expert replication communication to minimize exposed overhead. To this end, UltraEP eagerly reacts to post-gating load with an efficient quota-driven planner, and executes the resulting irregular expert-state transfers with RSN-native persistent tile streaming and relay-based fan-out mitigation. We evaluate UltraEP in a multi-RSN deployment of up to 256 GPUs, using cutting-edge MoE models from 106B to 671B parameters. Averaged across training and serving, UltraEP achieves 94.3% of the force-balanced ideal throughput, delivering 1.49

\times

improvement over no-balancing, while reducing the final inter-rank imbalance from 1.30

-

4.01 to 1.01

-

1.04.

Momentum

Related topics

Latest papers 55