cs.MANov 5, 2026

s3^3: Simple Scaling of Swarms via Cooperative Multi-agent RL

Authors: Kunal Jha, Gonçalo Faria, Shuyue Stella Li, Scott Geng, Faeze Brahman, Noah A. Smith, Yulia Tsvetkov, Max Kleiman-Weiner, 

Organizations: University of Washington. · Allen Institute for Artificial Intelligence.

Abstract

Groups can achieve more than individuals in the same amount of time, but only if their members coordinate: they must decide who works on what and what to share. However, this organization in multi-agent systems today usually comes from arbitrary, static harnesses designed a priori without adapting to agent behaviors or task specifications. We ask whether agents can instead learn to organize themselves. We introduce s3^3, the first multi-agent reinforcement learning method in which agents decide for themselves how long to work alone and when to share findings or read teammates' messages, with no predefined organization or roles. Our s3^3 agents learn to take different approaches, read each other's work, exhibit emergent roles that complement each other in the swarm, and generalize to bigger teams. We study both coordination, where agents must combine private information, and parallel exploration, where sharing discoveries can make search more efficient. With matched training and inference budgets, s3^3, swarms evaluated with twice as many agents as in training solve over 25×\times more coordination problems than the strongest baseline, and in exploration, communicating agents reduce repeated work by nearly 50% and outperform best-of-N by up to 9.4 percentage points. Rather than being designed in advance, s3^3 enables organization to emerge from a shared goal. This opens the door to capable systems built from many models that learn to work together.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Sep 28, 2026

Waggle: Learning One Anonymous Local Law for Self-Organizing LLM Swarms

As LLM agents increasingly collaborate on complex tasks, how to organize their interactions becomes a central design question. Existing multi-agent systems typically learn or adapt explicit roles, hierarchies, routing policies, or communication topologies. We shift the learning target to a reusable local law that can be shared across interchangeable agents and adapt coordination as populations or interaction conditions change, without redefining a global organization. We introduce Waggle, a shared anonymous policy over bounded local views that jointly selects task actions, semantic communication, and local commitment updates. Repeated execution of the same law allows coordination to form, persist, and reorganize online without explicit roles or global topology. To learn this law across interchangeable agents and evolving coordination, we develop Swarm-Consistent Distillation (SCD), combining anonymous-orbit consistency with rollout-grounded prediction of the next local coordination field, with no added inference-time components. Across diverse coordination settings, the same learned law remains effective as populations and interaction budgets change, retains over 96% of substrate-specific oracle quality, and transfers without retraining; SCD further improves reorganization after counterevidence. Together, these results show that LLM-agent organization can emerge and adapt through repeated execution of a learned local law.
Apr 1, 2026

OrgAgent: Organize Your Multi-Agent System like a Company

While large language model-based multi-agent systems have shown strong potential for complex reasoning, how to effectively organize multiple agents remains an open question. In this paper, we introduce OrgAgent, a company-style hierarchical multi-agent framework that separates collaboration into governance, execution, and compliance layers. OrgAgent decomposes multi-agent reasoning into three layers: a governance layer for planning and resource allocation, an execution layer for task solving and review, and a compliance layer for final answer control. By evaluating the framework across reasoning tasks, LLMs, execution modes, and execution policies, we find that multi-agent systems organized in a company-style hierarchy generally outperform other organizational structures. Besides, hierarchical coordination also reduces token consumption relative to flat collaboration in most settings. For example, for GPT-OSS-120B, the hierarchical setting improves performance over flat multi-agent system by 102.73% while reducing token usage by 74.52% on SQuAD 2.0. Further analysis shows that hierarchy helps most when tasks benefit from stable skill assignment, controlled information flow, and layered verification. Overall, our findings highlight organizational structure as an important factor in multi-agent reasoning, shaping not only effectiveness and cost, but also coordination behavior.
Oct 18, 2026

A learner-free swarm approximates the explore-exploit equilibrium allocation: A self-organized attractor in place of reinforcement-learning machinery

Reinforcement learning (RL) solves the explore-exploit problem by training a learner on reward; its multi-agent, resource-sharing form has an equilibrium allocation as its optimum -- the ideal-free distribution (IFD), the Nash equilibrium of a congestion game. We show a minimal swarm approximates it with no learner at all. We present a decentralized, learner-free method for this problem: agents explore by heavy-tailed (Levy) flights, recruit neighbours to a discovery through a degree-normalized threshold cascade, and disperse as a patch crowds. The method drives the collective to the shape of the equilibrium allocation (pooled spatial matching slope 1.14, R^2=0.61) with no training, no stored parameters, and no reward in its dynamics; it reaches this as a self-organized attractor and holds it, re-finding it when the resources move -- though it reaches the shape, not the exact equilibrium, and stays measurably exploitable (Nash-gap 1.46). Coupling recruitment to movement causes the allocation: removing it collapses the matching (slope 0.006). Against Proximal Policy Optimization (PPO) learners (feedforward, centralized-critic, graph message-passing, and recurrent) trained in the same sensing-limited environment, the method at zero training is the only arm left unbiased w.r.t. the IFD gradient (slope ~1, CI containing 1, vs learners 0.69-0.97, all under-concentrating) and ties the best learner's harvest where resource is clustered, while the learner collects more where resource is scarce (0.33 vs 0.22) and holds a tighter per-capita intake. Learning buys precision and transfer across reward structures, not a better allocation shape. A companion paper develops the mechanism.