Multi-agent systems (MAS) can outperform individual language-model agents by combining specialized roles, diverse reasoning processes, and collaborative decision-making. However, collaborative MAS training is typically centralized, requiring a common trainer to aggregate role-local trajectories and learning signals and maintain model and optimization states. This motivates a distributed multi-agent learning paradigm in which agents improve system-level capabilities through role-local learning and decentralized communication, paving the way toward scalable, resilient, and self-organizing MAS. To this end, we introduce, to the best of our knowledge, the first optimization-grounded framework for distributed MAS training based on consensus alternating direction method of multipliers (ADMM). We formulate collaborative MAS learning as a consensus-constrained finite-sum optimization problem, naturally lending itself to distributed optimization via ADMM. The proposed approach alternates role-local learning with inter-agent consensus using distributed communication protocols such as ring all-reduce, enabling joint optimization while keeping each agent's data and training state local. We evaluate our framework on distributed safety alignment and general reasoning tasks. Under compromised-agent settings, ADMM can reduce HarmBench attack success rate (ASR) by up to 26.4%, while achieving reasoning gains comparable to centralized training. These results show that distributed MAS training retains the benefits of centralized joint optimization without requiring a centralized trainer.
Figures & tables
Figure 1: Schematic overview of centralized and distributed MAS training. (a) Centralized training jointly optimizes all agents in a single training process. (b) Distributed training via ADMM performs role-local optimization with parameter exchange for inter-agent consensus. (c) Demonstration of safety training in MAS: Safety and utility of Qwen3-8B - based MAS (w/o training), centralized -trained, and ADMM -based distributed-trained MAS under two structures: sequential (S-MAS; Solver → Verifier → Corrector → Decider) and hierarchical (H-MAS; three Analyzers → Synthesizer). HarmBench attack success rate (ASR; ↓ , %) is reported for safety under benign (all agents safety-aligned) and compromised settings (unaligned Solver in S-MAS or Synthesizer in H-MAS), together with MMLU-Pro accuracy and mean accuracy across six reasoning benchmarks (Acc; ↑ , %). ADMM closely matches centralized training while substantially reducing HarmBench ASR relative to the base MAS. See Sec. 5 for more details.