s: Simple Scaling of Swarms via Cooperative Multi-agent RL
Organizations: University of Washington. · Allen Institute for Artificial Intelligence.
Abstract
Groups can achieve more than individuals in the same amount of time, but only if their members coordinate: they must decide who works on what and what to share. However, this organization in multi-agent systems today usually comes from arbitrary, static harnesses designed a priori without adapting to agent behaviors or task specifications. We ask whether agents can instead learn to organize themselves. We introduce s, the first multi-agent reinforcement learning method in which agents decide for themselves how long to work alone and when to share findings or read teammates' messages, with no predefined organization or roles. Our s agents learn to take different approaches, read each other's work, exhibit emergent roles that complement each other in the swarm, and generalize to bigger teams. We study both coordination, where agents must combine private information, and parallel exploration, where sharing discoveries can make search more efficient. With matched training and inference budgets, s, swarms evaluated with twice as many agents as in training solve over 25 more coordination problems than the strongest baseline, and in exploration, communicating agents reduce repeated work by nearly 50% and outperform best-of-N by up to 9.4 percentage points. Rather than being designed in advance, s enables organization to emerge from a shared goal. This opens the door to capable systems built from many models that learn to work together.
Figures & tables
| Condition | Description |
|---|---|
| Solo (SFT+RL) | Train a single agent with individual reward; evaluate independent copies as best-of- |
| s 3 communication | Team reward with messaging tools disabled |
| s 3 : individual reward | Messaging enabled; each agent is rewarded for its own success |
| s 3 : online SFT | Train on successful communicating teams sampled from the current policy |
| SFT only / initial | Fine-tuned checkpoint or base model, without RL |
| s 3 SFT | Team reward and messaging, trained from the base model |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Tool | Description given to the model |
|---|---|
| message(to, text) | “Send plans, questions, tentative ideas, dead ends, or checked results to one available teammate or all available teammates. Label uncertainty and verification status. Delivery is immediate. Sending does not read incoming messages or wait for replies; use read_messages when notified.” to is a teammate’s name or all . |
| read_messages() | “Read explicitly received unread peer messages.” Returns up to 16 unread messages with their senders. |
| check_notifications() | “Check unread message counts by sender without reading messages.” The harness also pushes this tool’s output on every arrival. |
| move(room) (decentralized exploration) | “Walk through a door of your current room into the named adjacent room. Returns what you see there. Your position is private: moving never moves a teammate.” |
| look() (decentralized exploration) | “Describe your current room again without moving.” |
| SFT on demonstrations | RL and evaluation | |
| Depth | 2 | 3 |
| Rooms | 121 | 391 |
| Dead ends (exit candidates) | 9 | 27 |
| Moves on the shortest path | 20 | 30 |
| Moves for a full depth-first search | 240 | 780 |
| Training / development / evaluation search trees | 1,024 / 32 / 32 | 1,024 / 32 / 32 |
| Setting | Information aggregation | Decentralized exploration |
|---|---|---|
| Model | Qwen3-0.6B, thinking on | Qwen3-1.7B, thinking on, all reasoning retained |
| Initial policy | Base model | 50 SFT updates on demonstrations (Appendix L ) |
| RL updates | 200 | 200 |
| Teams per update | 256 (32 problems 8 attempts) | 128 (16 problems 8 attempts) |
| Team size in training | 4 | 4 |
| Per-agent allowance | 8,192 work + 1,024 final | 8,192 (+1 unused final token) |
| Learner | Problems | Episodes | Trajectories | Adv. group | Loss unit |
|---|---|---|---|---|---|
| s 3 | 16 | 128 teams of 4 | 512 | 8 teams | team |
| s 3 communication | 16 | 128 teams of 4 | 512 | 8 teams | team |
| s 3 : individual reward | 16 | 128 teams of 4 | 512 | 32 agents | team |
| Solo learner | 16 | 512 single agents | 512 | 32 agents | agent |
| Reward | Success (%) | Efficiency | |||||
| Learner | Start | End | Start | End | (pp) | Start | End |
| Sorting | |||||||
| s 3 (Ours) | +90.3 | ||||||
| s 3 : individual reward | +5.4 | ||||||
| s 3 : online SFT | +25.9 | ||||||
| Mazes | |||||||
| Learner | Start success (%) | Final success (%) | Gain (pp) |
|---|---|---|---|
| Exploration: s 3 (Ours) | |||
| Exploration: Solo (SFT+RL) | |||
| Exploration: s 3 communication | |||
| Exploration: s 3 : individual reward | |||
| Exploration: s 3 SFT | |||
| Exploration: Solo RL (no SFT) |
| Team learner | Success gain (pp) | Efficiency gain | |
|---|---|---|---|
| s 3 (Ours) | 1 | ||
| s 3 (Ours) | 2 | ||
| s 3 (Ours) | 4 | ||
| s 3 (Ours) | 6 | ||
| s 3 (Ours) | 8 | ||
| s 3 (Ours) | 12 |
| Family / task | Contrast | Metric | Effect SE | Before | After | ||
|---|---|---|---|---|---|---|---|
| Train / expl. | s 3 vs. Solo (SFT+RL) | 4 | Success | 1.22 | Yes | No (2) | |
| Train / expl. | s 3 vs. Solo (SFT+RL) | 4 | Efficiency | 0.23 | No | No (2) | |
| Scale / expl. | s 3 vs. Solo (SFT+RL) | 6 | Success | 1.71 | Yes | Yes (2) | |
| Scale / expl. | s 3 vs. Solo (SFT+RL) | 6 | Efficiency | 0.00 | No | No (2) | |
| Train / expl. | s 3 SFT vs. Solo RL (no SFT) | 4 | Success | -0.48 | No | No (2) | |
| Abl. / expl. | s 3 vs. s 3 : individual reward | 4 | Success | -0.28 | No | No (3) |
| s 3 | |||
|---|---|---|---|
| Structure | 1–100 | 101–200 | Share |
| An agent never messages | 8% | ||
| Conversation (reference) | 0 | 0 | 18% |
| Repeated broadcasts | 26% | ||
| One broadcast each | 48% | ||
| Updates 1–20 | s 3 , 101–200 | s 3 : individual reward, 101–200 | s 3 : online SFT, 101–200 | |
| Message content | ||||
| Own number (“My private number is forty-two.”) | 71% | 90% | 64% | 74% |
| Request only (“What is your private number?”) | 21% | 3% | 26% | 16% |
| Own number stated as recipient’s | 4% | 1% | 6% | 3.5% |
| Misstated own number | 1.6% | 1.2% | 1.7% | 1.9% |
| Failure cause | ||||