Distributed Optimization

Follow this topic Ask about this topic

Distributed optimization studies how multiple machines, clients, or robots can solve a shared problem using local data and limited communication. Common approaches include consensus and decentralized gradient methods, local or federated SGD, ADMM and primal-dual algorithms, and asynchronous updates; communication can be reduced through compression, quantization, or less frequent synchronization. These methods support settings from large-model training and statistical learning to robot coordination, energy systems, and networked control. A central challenge is balancing communication and computation against convergence when data and worker speeds differ, messages are delayed, or participants may be unreliable or adversarial.

Latest papers

92 papers11 papers in the last four weeks, up 267% on the four weeks before. 0.1% of all new papers.

Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure
Jun 24, 2026Maxwell Twelftree, David Lemphers, An-chi He +1Communication-Efficient Distributed Training Distributed Optimization
Maincode, Melbourne, Australia
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Jun 22, 2026Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7Communication-Efficient Distributed Training Distributed Optimization
1Pluralis Research
Deep-Unfolded Coordination
Jun 18, 2026Hunter Kuperman, Minchan Jung, Rahul V. Ghosh +2Trajectory Optimization Multi-Agent System Optimization
Autonomous Control and Decision Systems Laboratory Georgia Institute of Technology United States
Semiglobal Input-Delay Tolerance Algorithm for Distributed Nonconvex Optimization of Networked Nonlinear Systems
Jun 18, 2026Jing-Zhe Xu, Zhi-Wei Liu, Ming-Feng Ge +2Dynamical Systems Optimization Convergence Analysis
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China · School of Mechanical Engineering and Electronic Information, China University of Geosciences, Wuhan 430074, China
Quantized Stochastic Primal-Dual Methods for Distributed Optimization under Relaxed Global Geometry
Jun 9, 2026Susmit Sarkar, Abhinav Raghuvanshi, Kushal Chakrabarti +1Stochastic Optimization Primal-Dual Optimization
Indian Institute of Technology Bombay, Mumbai, Maharashtra, India · Tata Consultancy Services Research, Thane, Maharashtra, India
Piper: A Programmable Distributed Training System
Jun 9, 2026Megan Frisella, Shubham Tiwari, Andy Ruan +5Distributed Optimization Distributed Training
University of Washington Seattle, WA, USA · University of Washington and Shanghai Jiao Tong University Seattle, WA, USA
Unifying Local Communications and Local Updates for LLM Pretraining
Jun 9, 2026Pietro Cagnasso, Eugene Belilovsky, Edouard OyallonLanguage Model Pretraining Communication-Efficient Distributed Training
Concordia University · Mila · CNRS, Sorbonne University
A Note on Stability for Orthogonalized Matrix Momentum with Client Sampling
Jun 1, 2026Da Chang, Qiankun Shi, Lvgang Zhang +2Algorithmic Stability Distributed Optimization
University of Chinese Academy of Sciences · Sun Yat-sen University · Southern University of Science and Technology +1
Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning
May 31, 2026Tehila Dahan, Bassel Hamoud, Roie Reshef +2Communication-Efficient Distributed Training Distributed Optimization
Technion Haifa, Israel · EPFL Lausanne, Switzerland
Distributed GNEP Algorithms without Multiplier Sharing and Applications to Multi-Robot Coordination and Contextual Bandit-Based Active Learning
May 30, 2026Shao-An YinGame Theory Distributed Optimization
UNIVERSITY OF MINNESOTA
A Tight Theory of Error Feedback Algorithms in Distributed Optimization
May 29, 2026Daniel Berg Thomsen, Adrien Taylor, Aymeric DieuleveutCommunication-Efficient Distributed Training Distributed Optimization
Inria, D.I. ENS, CNRS, PSL Research University, Paris, France · CMAP, École Polytechnique, Institut Polytechnique de Paris, Palaiseau, France
Scalable Constrained Multi-Agent Reinforcement Learning via State Augmentation and Consensus for Separable Dynamics
May 28, 2026Santiago Amaya-Corredor, Miguel Calvo-Fullana, Anders JonssonMulti-Agent System Optimization Multi-Agent Coordination
Department of Engineering, University Pompeu Fabra
Outer-Momentum Restarting in High-Dimensional Two-Phase Optimization
May 27, 2026Kristi Topollai, Allan Ma, Tolga Dimlioglu +2Momentum Methods Communication-Efficient Distributed Training
New York University, New York, USA
DEI: Diversity in Evolutionary Inference for Quality-Diversity Search
May 26, 2026John Donaghy, Shikhar RastogiDistributed Optimization Large Language Model-Guided Optimization
Gensyn
A Generalized Nash Equilibrium-Seeking Scheme for Trauma Resuscitation
May 21, 2026Promise Ekpo, Angelique Taylor, Lekan MoluNash Equilibrium Multi-Agent Coordination
Cornell University, NY. · Bala Cynwyd, PA.
Privacy-Preserving Distributed Optimization Under Time Constraints Using Secure Multi-Party Computation and Evolutionary Algorithms
May 20, 2026Sebastian Gruber, Tobias Harzfeld, Christoph G. Schuetz +2Secure Multi-Party Computation Distributed Optimization
Johannes Kepler University Linz, Linz, Austria · AIT Austrian Institute of Technology, Vienna, Austria
LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging
May 20, 2026Yassine Maziane, Ammar Mahran, Artavazd Maranjyan +1Communication-Efficient Distributed Training Distributed Optimization
KAUST
Efficient Gradient Methods for Distributed Saddle Problems
May 18, 2026Ruichen Luo, Anton Rodomanov, Sebastian U. StichDistributed Optimization Decentralized Optimization
Institute of Science and Technology Austria · CISPA Helmholtz Center
Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Method
May 18, 2026Abdurakhmon Sadiev, Artavazd Maranjyan, Ivan Ilin +1LMO Methods Momentum Methods
KAUST
A Prototyping Framework for Distributed Control of Multi-Robot Systems
May 14, 2026Junaid Ahmed Memon, Allan Andre Do Nascimento, Kostas Margellos +1Multi-Robot Systems Robotics Simulation
Department of Engineering Science, University of Oxford, UK
Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity
May 13, 2026Ammar Mahran, Artavazd Maranjyan, Peter RichtárikDistributed Optimization Nonconvex Stochastic Optimization
KAUST
Byzantine-Robust Distributed Sparse Learning Revisited
May 13, 2026Yuxuan Wang, Lixin Zhang, Kangqiang LiDistributed Training Distributed Optimization
School of Mathematical Sciences, Zhejiang University, Hangzhou, 310027, Zhejiang, China · School of Statistics and Mathematics, Zhejiang Gongshang University, Hangzhou, 310018, Zhejiang, China · Information Center, Hubei Provincial Tobacco Monopoly Administration, Wuhan, 430030, Hubei, China
Distributed Pose Graph Optimization via Continuous Riemannian Dynamics
May 11, 2026Jaeho Shin, Maani Ghaffari, Yulun TianRiemannian Optimization Multi-Robot Systems
University of Michigan Ann Arbor, MI 48109, USA
Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimates
May 10, 2026Anik Kumar Paul, Nibedita Roy, Nagesh Talagani +3Stochastic Optimization Sign-Based Optimization
Computer Science and Automation, Indian Institute of Science, Bengaluru · Edwardson School of Industrial Engineering, Purdue University · Department of Computer Science, IMT Atlantique
Cosine-Gated Adam-Decay: Drop-In Staleness-Aware Outer Optimization for Decoupled DiLoCo
May 9, 2026Vatsal Shah, Jiahao SunCommunication-Efficient Distributed Training Distributed Optimization
Rennala MVR: Improved Time Complexity for Parallel Stochastic Optimization via Momentum-Based Variance Reduction
May 9, 2026Zhirayr Tovmasyan, Artavazd Maranjyan, Peter RichtárikStochastic Optimization Distributed Optimization
KAUST
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
May 7, 2026Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5Communication-Efficient Distributed Training Distributed Optimization
1Keio University · 2Stanford University · 3Midwestern University +3
Distributed Learning with Adversarial Gradient Perturbations
May 5, 2026Nawapon Sangsiri, Yufei TaoDistributed Optimization Adversarial Perturbations
CUHK
SignMuon: Communication-Efficient Distributed Muon Optimization
May 4, 2026Neel Mishra, Jichao Jiang, Kushagara Trivedi +3Sign-Based Optimization Communication-Efficient Distributed Training
Microsoft · IIIT Hyderabad
Bringing Order to Asynchronous SGD: Towards Optimality under Data-Dependent Delays with Momentum
May 3, 2026Tehila Dahan, Roie Reshef, Sharon Goldstein +1Momentum Methods Distributed Optimization
Technion – Israel Institute of Technology, Haifa, Israel

Jun 24, 2026J/K move · Enter open · S save

Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure

Maxwell Twelftree, David Lemphers, An-chi He, Yue Yang

Maincode, Melbourne, Australia

Communication-Efficient Distributed Training Distributed Optimization Distributed Training

Open paper AskPDF arXiv

DiLoCo-style training reduces communication by letting learner islands train locally before occasional outer synchronization, making it attractive for fragmented industrial AI fleets where training shares hardware with latency-sensitive serving. The question for such fleets is when an outer merge is worth its system cost, and whether choosing which windows to defer matters at all. Existing scheduling studies evaluate workload-aware policies against fixed-period baselines, but most omit the control that isolates timing from budget: matched random deferral, which inherits the controller's synchronization budget but is not itself deployable. This omission is consequential: across controlled stress tests and real vLLM sidecar replays, matched random ties or beats every forecast-free policy we test, so gains reported against weaker baselines cannot be attributed to window choice. We fill this gap with Workload-Aware DiLoCo (WA-DiLoCo), a score-based controller that weighs learner progress against fleet pressure, and a calibration protocol that determines when matched random can be beaten, then demonstrate that it can. In the bursty regime where calibration exposes request-overlap structure, adding a one-step EWMA burst forecast to the online controller beats matched random in real vLLM sidecar replay, reducing SLO violations from 6.54% to 5.09% (8 of 10 seeds,

p=0.021

); offline Calibrated-WA, a non-deployable bound, shows the remaining headroom at 4.45% versus 6.26%. The deployable lesson remains the protocol: report real-sidecar effect-size transfer, a no-sync load match, and a matched-random envelope before claiming serving-SLO improvement.